The LMU Munich System for the WMT 2020 Unsupervised Machine Translation Shared Task

Proceedings of the 5th Conference on Machine Translation (WMT), pages 1084–1091 Online, November 19–20, 2020. c⃝2020 Association for Computational Linguistics 1084 The LMU Munich System for the WMT 2020 Unsupervised Machine Translation Shared Task Alexandra Chronopoulou, Dario Stojanovski, Viktor Hangya, Alexander Fraser Center for Information and Language Processing, LMU Munich, Germany {achron, stojanovski, hangyav, fraser}@cis.lmu.de Abstract This paper describes the submission of LMU Munich to the WMT 2020 unsupervised shared task, in two language directions, German↔Upper Sorbian. Our core unsuper- vised neural machine translation (UNMT) sys- tem follows the strategy of Chronopoulou et al. (2020), using a monolingual pretrained lan- guage generation model (on German) and fine- tuning it on both German and Upper Sorbian, before initializing a UNMT model, which is trained with online backtranslation. Pseudo- parallel data obtained from an unsupervised statistical machine translation (USMT) system is used to fine-tune the UNMT model. We also apply BPE-Dropout to the low-resource (Up- per Sorbian) data to obtain a more robust sys- tem. We additionally experiment with resid- ual adapters and find them useful in the Up- per Sorbian→German direction. We explore sampling during backtranslation and curricu- lum learning to use SMT translations in a more principled way. Finally, we ensemble our best- performing systems and reach a BLEU score of 32.4 on German→Upper Sorbian and 35.2 on Upper Sorbian→German. 1 Introduction Neural machine translation achieves remarkable re- sults (Bahdanau et al., 2015; Vaswani et al., 2017) when large parallel training corpora are available. However, such corpora are only available for a limited number of languages. UNMT addresses this issue by using monolingual data only (Artetxe et al., 2018c; Lample et al., 2018). The perfor- mance of UNMT models is further improved using transfer learning from a pretrained cross-lingual model (Lample and Conneau, 2019; Song et al., 2019). However, pretraining also demands large monolingual corpora for both languages. Without abundant data, UNMT methods are often ineffective (Guzm´an et al., 2019). Therefore, effectively trans- lating between a high-resource and a low-resource language, in terms of monolingual data, which is the target of this year’s unsupervised shared task, is challenging. We participate in the WMT 2020 unsuper- vised machine translation shared task. The task includes two directions: German→Upper Sor- bian (De→Hsb) and Upper Sorbian→German (Hsb→De). Our systems are constrained, using only the provided Hsb monolingual data and De NewsCrawl monolingual data released for WMT. We pretrain a monolingual encoder-decoder model on a language generation task with the Masked Se- quence to Sequence model (MASS) (Song et al., 2019) and fine-tune it on both languages of interest, following Chronopoulou et al. (2020). We then train it on UNMT, using online backtranslation. We use our USMT system to backtranslate monolin- gual data in both languages. This pseudo-parallel corpus serves to fine-tune our UNMT model. Itera- tive offline backtranslation is later leveraged, yield- ing a performance boost. We use BPE-Dropout (Provilkov et al., 2020) as a data augmentation technique, sampling instead of greedy decoding in online backtranslation, and curriculum learning to best include the SMT …

Content truncated to 500 words