Reusing a Pretrained Language Model on Languages with Limited Corpora for Unsupervised NMT

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 2703–2711, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 2703 Reusing a Pretrained Language Model on Languages with Limited Corpora for Unsupervised NMT Alexandra Chronopoulou, Dario Stojanovski, Alexander Fraser Center for Information and Language Processing, LMU Munich, Germany {achron, stojanovski, fraser}@cis.lmu.de Abstract Using a language model (LM) pretrained on two languages with large monolingual data in order to initialize an unsupervised neural ma- chine translation (UNMT) system yields state- of-the-art results. When limited data is avail- able for one language, however, this method leads to poor translations. We present an ef- fective approach that reuses an LM that is pretrained only on a high-resource language. The monolingual LM is fine-tuned on both lan- guages and is then used to initialize a UNMT model. To reuse the pretrained LM, we have to modify its predefined vocabulary, to ac- count for the new language. We therefore propose a novel vocabulary extension method. Our approach, RE-LM, outperforms a com- petitive cross-lingual pretraining model (XLM) in English-Macedonian (En-Mk) and English- Albanian (En-Sq), yielding more than +8.3 BLEU points for all four translation directions. 1 Introduction Neural machine translation (NMT) has recently achieved remarkable results (Bahdanau et al., 2015; Vaswani et al., 2017), based on the exploitation of large parallel training corpora. Such corpora are only available for a limited number of languages. UNMT has attempted to address this limitation by training NMT systems using monolingual data only (Artetxe et al., 2018; Lample et al., 2018). Top performance is achieved using a bilingual masked language model (Devlin et al., 2019) to initial- ize a UNMT encoder-decoder system (Lample and Conneau, 2019). The model is then trained us- ing denoising auto-encoding (Vincent et al., 2008) and back-translation (Sennrich et al., 2016a). The approach was mainly evaluated by translating be- tween high-resource languages. Translating between a high-resource and a low- resource language is a more challenging task. In this setting, the UNMT model can be initialized with a pretrained cross-lingual LM. However, training this UNMT model has been shown to be ineffective when the two languages are not related (Guzmán et al., 2019). Moreover, in order to use a pretrained cross-lingual LM to initialize a UNMT model, the two models must have a shared vocabulary. Thus, a bilingual LM needs to be trained from scratch for each language pair, before being transferred to the UNMT model (e.g. En-De LM for En-De UNMT). Motivated by these issues, we focus on the ques- tion: how can we accurately and efficiently trans- late between a high-monolingual-resource (HMR) and a low-monolingual-resource (LMR) language? To address this question, we adapt a monolingual LM, pretrained on an HMR language to an LMR language, in order to initialize a UNMT system. We make the following contributions: (1) We propose REused-LM1 (RE-LM), an effective trans- fer learning method for UNMT. Our method reuses a pretrained LM on an HMR language, by fine- tuning it on both LMR and HMR languages. The fine-tuned …

Content truncated to 500 words