A Multilingual View of Unsupervised Machine Translation

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3160–3170 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 3160 A Multilingual View of Unsupervised Machine Translation Xavier Garcia˚ {xgarcia,pierreforet,tsellam,aparikh}@google.com Pierre Foret˚ Thibault Sellam Ankur P. Parikh Abstract We present a probabilistic framework for mul- tilingual neural machine translation that en- compasses supervised and unsupervised se- tups, focusing on unsupervised translation. In addition to studying the vanilla case where there is only monolingual data available, we propose a novel setup where one language in the (source, target) pair is not associated with any parallel data, but there may exist auxiliary parallel data that contains the other. This auxil- iary data can naturally be utilized in our proba- bilistic framework via a novel cross-translation loss term. Empirically, we show that our ap- proach results in higher BLEU scores over state-of-the-art unsupervised models on the WMT’14 English-French, WMT’16 English- German, and WMT’16 English-Romanian datasets in most directions. 1 Introduction The popularity of neural machine translation sys- tems (Kalchbrenner and Blunsom, 2013; Sutskever et al., 2014; Bahdanau et al., 2015; Wu et al., 2016) has exploded in recent years. Those systems have obtained state-of-the-art results for a wide col- lection of language pairs, but they often require large amounts of parallel (source, target) sentence pairs to train (Koehn and Knowles, 2017), mak- ing them impractical for scenarios with resource- poor languages. As a result, there has been in- terest in unsupervised machine translation (Ravi and Knight, 2011), and more recently unsuper- vised neural machine translation (UNMT) (Lample et al., 2018; Artetxe et al., 2018), which uses only monolingual source and target corpora for learning. Unsupervised NMT systems have achieved rapid progress recently (Lample and Conneau, 2019; Artetxe et al., 2019; Ren et al., 2019; Li et al., 2020a), largely thanks to two key ideas: one-the-fly back-translation (i.e., minimizing round-trip trans- lation inconsistency) (Bannard and Callison-Burch, ˚ Work done as part of the Google AI Residency. Fr En Ro (a) Multilingual NMT En Fr Ro (b) Zero-Shot NMT En Fr Ro (c) M-UNMT (w/o auxiliary parallel data) En Fr Ro (d) M-UNMT (w/ auxiliary parallel data) Figure 1: Different setups for English (En), French (Fr) and Romanian (Ro). The dashed edge indicates the target language pair. Full edges indicate the exis- tence of parallel training data. 2005; Sennrich et al., 2015; He et al., 2016; Artetxe et al., 2018) and pretrained language models (Lam- ple and Conneau, 2019; Song et al., 2019). Despite the difficulty of the problem, those systems have achieved surprisingly strong results. In this work, we investigate Multilingual UNMT (M-UNMT), a generalization of the UNMT setup that involves more than two languages. Multilin- guality has been explored in the supervised NMT literature, where it has been shown to enable in- formation sharing among related languages. This allows higher resource language pairs (e.g. English– French) to improve performance among lower re- source pairs (e.g., English–Romanian) (Johnson et al., 2017; Firat et al., 2016). Yet multilingual translation has only received little …

Content truncated to 500 words