Massively Multilingual Neural Machine Translation

Proceedings of NAACL-HLT 2019, pages 3874–3884 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 3874 Massively Multilingual Neural Machine Translation Roee Aharoni∗ Bar Ilan University Ramat-Gan Israel roee.aharoni@gmail.com Melvin Johnson and Orhan Firat Google AI Mountain View California melvinp,orhanf@google.com Abstract Multilingual neural machine translation (NMT) enables training a single model that supports translation from multiple source lan- guages into multiple target languages. In this paper, we push the limits of multilingual NMT in terms of the number of languages being used. We perform extensive experiments in training massively multilingual NMT models, translating up to 102 languages to and from English within a single model. We explore different setups for training such models and analyze the trade-offs between translation quality and various modeling decisions. We report results on the publicly available TED talks multilingual corpus where we show that massively multilingual many-to-many models are effective in low resource settings, outperforming the previous state-of-the-art while supporting up to 59 languages. Our experiments on a large-scale dataset with 102 languages to and from English and up to one million examples per direction also show promising results, surpassing strong bilingual baselines and encouraging future work on massively multilingual NMT. 1 Introduction Neural machine translation (NMT) (Kalchbren- ner and Blunsom, 2013; Bahdanau et al., 2014; Sutskever et al., 2014) is the current state-of- the-art approach for machine translation in both academia (Bojar et al., 2016, 2017, 2018) and in- dustry (Wu et al., 2016; Hassan et al., 2018). Re- cent works (Dong et al., 2015; Firat et al., 2016a; Ha et al., 2016; Johnson et al., 2017) extended the approach to support multilingual translation, i.e. training a single model that is capable of translat- ing between multiple language pairs. Multilingual models are appealing for several reasons. First, they are more efficient in terms ∗Work carried out during an internship at Google AI. of the number of required models and model pa- rameters, enabling simpler deployment. Another benefit is transfer learning; when low-resource language pairs are trained together with high- resource ones, the translation quality may improve (Zoph et al., 2016; Nguyen and Chiang, 2017). An extreme case of such transfer learning is zero-shot translation (Johnson et al., 2017), where multilin- gual models are able to translate between language pairs that were never seen during training. While very promising, it is still unclear how far one can scale multilingual NMT in terms of the number of languages involved. Previous works on multilingual NMT typically trained models with up to 7 languages (Dong et al., 2015; Fi- rat et al., 2016b; Ha et al., 2016; Johnson et al., 2017; Gu et al., 2018) and up to 20 trained direc- tions (Cettolo et al., 2017) simultaneously. One recent exception is Neubig and Hu (2018) who trained many-to-one models from 58 languages into English. While utilizing significantly more languages than previous works, their experiments were restricted to many-to-one models in a low- resource setting with up to 214k examples per language-pair and were evaluated …

Content truncated to 500 words