Harnessing Multilinguality in Unsupervised Machine Translation for Rare Languages

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1126–1137 June 6–11, 2021. ©2021 Association for Computational Linguistics 1126 Harnessing Multilinguality in Unsupervised Machine Translation for Rare Languages Xavier Garcia and Aditya Siddhant and Orhan Firat and Ankur P. Parikh Google Research Mountain View California {xgarcia,adisid,orhanf,aparikh}@google.com Abstract Unsupervised translation has reached impres- sive performance on resource-rich language pairs such as English-French and English- German. However, early studies have shown that in more realistic settings involving low- resource, rare languages, unsupervised transla- tion performs poorly, achieving less than 3.0 BLEU. In this work, we show that multilin- guality is critical to making unsupervised sys- tems practical for low-resource settings. In particular, we present a single model for 5 low- resource languages (Gujarati, Kazakh, Nepali, Sinhala, and Turkish) to and from English di- rections, which leverages monolingual and aux- iliary parallel data from other high-resource lan- guage pairs via a three-stage training scheme. We outperform all current state-of-the-art unsu- pervised baselines for these languages, achiev- ing gains of up to 14.4 BLEU. Additionally, we outperform strong supervised baselines for various language pairs as well as match the per- formance of the current state-of-the-art super- vised model for NeÑEn. We conduct a series of ablation studies to establish the robustness of our model under different degrees of data quality, as well as to analyze the factors which led to the superior performance of the proposed approach over traditional unsupervised models. 1 Introduction Neural machine translation systems (Kalchbrenner and Blunsom, 2013; Sutskever et al., 2014; Bah- danau et al., 2015; Wu et al., 2016) have demon- strated state-of-the-art results for a diverse set of language pairs when given large amounts of rele- vant parallel data. However, given the prohibitive nature of such a requirement for low-resource lan- guage pairs, there has been a growing interest in unsupervised machine translation (Ravi and Knight, 2011) and its neural counterpart, unsupervised neu- ral machine translation (UNMT) (Lample et al., 2018a; Artetxe et al., 2018), which leverage only monolingual source and target corpora for learn- ing. Bilingual unsupervised systems (Lample and Conneau, 2019; Artetxe et al., 2019; Ren et al., 2019; Li et al., 2020a) have achieved surprisingly strong results on high-resource language pairs such as English-French and English-German. However, these works only evaluate on high- resource language pairs with high-quality data, which are not realistic scenarios where UNMT would be utilized. Rather, the practical potential of UNMT is in low-resource, rare languages that may not only lack parallel data but also have a shortage of high-quality monolingual data. For in- stance, Romanian (a typical evaluation language for unsupervised methods) has 21 million lines of high-quality in-domain monolingual data provided by WMT. In contrast, for an actual low-resource language, Gujarati, WMT only provides 500 thou- sand lines of monolingual data (in news domain) and an additional 3.7 million lines of monolingual data from Common Crawl (noisy, general-domain). Given the comparably sterile setups UNMT has been studied in, …

Content truncated to 500 words