The NITS-CNLP System for the Unsupervised MT Task at WMT 2020

Proceedings of the 5th Conference on Machine Translation (WMT), pages 1139–1143 Online, November 19–20, 2020. c⃝2020 Association for Computational Linguistics 1139 The NITS-CNLP System for the Unsupervised MT Task at WMT 2020 Salam Michael Singh Thoudam Doren Singh Center for Natural Language Processing (CNLP) and Dept. of Computer Science & Engg. National Institute of Technology Silchar, India {salammichaelcse,thoudam.doren,sivaji.cse.ju}@gmail.com Sivaji Bandyopadhyay Abstract We describe NITS-CNLP’s submission to WMT 2020 unsupervised machine translation shared task for German language (de) to Up- per Sorbian (hsb) in a constrained setting i.e, using only the data provided by the organiz- ers. We train our unsupervised model using monolingual data from both the languages by jointly pre-training the encoder and decoder and fine-tune using backtranslation loss. The final model uses the source side (de) monolin- gual data and the target side (hsb) synthetic data as a pseudo-parallel data to train a pseudo- supervised system which is tuned using the provided development set(dev set). 1 Introduction This paper provides the system description of the unsupervised neural machine translation system for German to Upper Sorbian submitted by the Center for Natural Language Processing of National Insti- tute of Technology, Silchar, India (NITS-CNLP) in the WMT 2020 shared task for Unsupervised and Very Low Resource machine translation for German and Upper-Sorbian language pair. Specif- ically, we made our primary submission for the unsupervised task in de → hsb direction. We use the data provided by the organisers only i.e, in a constrained manner. Our unsupervised neural ma- chine translation (UNMT) system first pre-trains a transformer (Vaswani et al., 2017) based encoder and decoder model using masked sequence to se- quence (MASS) pre-training (Song et al., 2019) and fine-tune using the back-translation (Sennrich et al., 2016a) loss. The final model trained using MASS objective is then used to translate the source side (Mde) monolingual data into a synthetic target side data (M ′ hsb)and then train a pseudo-supervised model using {Mde,M ′ hsb} from scratch. The remaining of the paper is arranged in fol- lowing manner: Section 2 gives a brief background of an unsupervised MT. Section 3 describes the data preprocessing. In Section 4, we describe our UNMT system. The results and analysis are shown in Section 5. Finally, Section 6 concludes the paper. 2 Background NMT (Kalchbrenner and Blunsom, 2013; Cho et al., 2014; Bahdanau et al., 2014) has become the de-facto MT system in recent times achiev- ing near human level translation quality for many language pair however at the cost of millions of bi-text data. Unfortunately, bi-text data for many languages is scarce or non-existent. Unsupervised MT (Lample et al., 2018a; Artetxe et al., 2018b) is one of the techniques to handle the bi-text un- availability by exploiting monolingual data (Sen- nrich et al., 2016a). Primitive unsupervised MT first maps the monolingual data into a common cross-lingual shared vector embedding space (Con- neau et al., 2017; Artetxe et al., 2017) and infer a bilingual dictionary from this shared space us- ing adversarial training (Lample et al., …

Content truncated to 500 words