Complete Multilingual Neural Machine Translation

Proceedings of the 5th Conference on Machine Translation (WMT), pages 550–560 Online, November 19–20, 2020. c⃝2020 Association for Computational Linguistics 550 Complete Multilingual Neural Machine Translation Markus Freitag and Orhan Firat Google Research {freitag,orhanf}@google.com Abstract Multilingual Neural Machine Translation (MNMT) models are commonly trained on a joint set of bilingual corpora which is acutely English-centric (i.e. English either as the source or target language). While direct data between two languages that are non-English is explicitly available at times, its use is not common. In this paper, we first take a step back and look at the commonly used bilingual corpora (WMT), and resurface the existence and importance of implicit structure that existed in it: multi-way alignment across examples (the same sentence in more than two languages). We set out to study the use of multi-way aligned examples to enrich the original English-centric parallel corpora. We reintroduce this direct parallel data from multi-way aligned corpora between all source and target languages. By doing so, the English-centric graph expands into a complete graph, every language pair being connected. We call MNMT with such connectivity pat- tern complete Multilingual Neural Machine Translation (cMNMT) and demonstrate its utility and efficacy with a series of experi- ments and analysis. In combination with a novel training data sampling strategy that is conditioned on the target language only, cMNMT yields competitive translation quality for all language pairs. We further study the size effect of multi-way aligned data, its transfer learning capabilities and how it eases adding a new language in MNMT. Finally, we stress test cMNMT at scale and demonstrate that we can train a cMNMT model with up to 111∗112=12,432 language pairs that provides competitive translation quality for all language pairs. 1 Introduction Multilingual machine translation (Dong et al., 2015; Firat et al., 2016a; Johnson et al., 2017; Aharoni et al., 2019), which can serve multiple (a) English-centric (b) Complete Figure 1: Source-target translation graphs in MNMT. Solid lines indicate that there exist direct parallel data. When there is no line connecting any two languages, zero-resource or zero-shot approaches are employed. language pairs with a single model, has attracted much attention. In contrast to bilingual MT sys- tems which can only serve one single language pair, multilingual models can serve O(N2) lan- guage pairs (N being the number of languages in a multilingual model) (Zhang et al., 2020). The amount of available training data can differ a lot across language pairs and the majority of avail- able MT training data is English-centric (Tiede- mann, 2018; Arivazhagan et al., 2019b) which in practice means that most non-English language pairs do not see a single training example when training multilingual models (see Figure 1a). As a consequence, the actual performance of language pairs that do not include English on the source or target side lags behind the ones with large amounts of training data. Further, when increasing the num- ber of languages, it gets (a) impractical to gather training data for each language pair and (b) chal- lenging …

Content truncated to 500 words