Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1628–1639 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 1628 Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation Biao Zhang1 Philip Williams1 Ivan Titov1,2 Rico Sennrich3,1 1School of Informatics, University of Edinburgh 2ILLC, University of Amsterdam 3Department of Computational Linguistics, University of Zurich B.Zhang@ed.ac.uk, {pwillia4,ititov}@inf.ed.ac.uk, sennrich@cl.uzh.ch Abstract Massively multilingual models for neural ma- chine translation (NMT) are theoretically at- tractive, but often underperform bilingual mod- els and deliver poor zero-shot translations. In this paper, we explore ways to improve them. We argue that multilingual NMT re- quires stronger modeling capacity to support language pairs with varying typological char- acteristics, and overcome this bottleneck via language-specific components and deepening NMT architectures. We identify the off-target translation issue (i.e. translating into a wrong target language) as the major source of the inferior zero-shot performance, and propose random online backtranslation to enforce the translation of unseen training language pairs. Experiments on OPUS-100 (a novel multilin- gual dataset with 100 languages) show that our approach substantially narrows the perfor- mance gap with bilingual models in both one- to-many and many-to-many settings, and im- proves zero-shot performance by ∼10 BLEU, approaching conventional pivot-based meth- ods.1 1 Introduction With the great success of neural machine transla- tion (NMT) on bilingual datasets (Bahdanau et al., 2015; Vaswani et al., 2017; Barrault et al., 2019), there is renewed interest in multilingual translation where a single NMT model is optimized for the translation of multiple language pairs (Firat et al., 2016a; Johnson et al., 2017; Lu et al., 2018; Aha- roni et al., 2019). Multilingual NMT eases model deployment and can encourage knowledge transfer among related language pairs (Lakew et al., 2018; Tan et al., 2019), improve low-resource transla- tion (Ha et al., 2016; Arivazhagan et al., 2019b), 1We release our code at https://github. com/bzhangGo/zero. We release the OPUS-100 dataset at https://github.com/EdinburghNLP/ opus-100-corpus. Source Jusqu’à ce qu’on trouve le moment clé, celui où tu as su que tu l’aimais. Reference Bis wir den unverkennbaren Moment gefun- den haben, den Moment, wo du wusstest, du liebst ihn. Zero-Shot Jusqu’à ce qu’on trouve le moment clé, celui où tu as su que tu l’aimais. Source Les États membres ont été consultés et ont approuvé cette proposition. Reference Die Mitgliedstaaten wurden konsultiert und sprachen sich für diesen Vorschlag aus. Zero-Shot Les Member States have been consultedés and have approved this proposal. Table 1: Illustration of the off-target translation issue with French→German zero-shot translations with a multilingual NMT model. Our baseline multilingual NMT model often translates into the wrong language for zero-shot language pairs, such as copying the source sentence or translating into English rather than German. and enable zero-shot translation (i.e. direct trans- lation between a language pair never seen in train- ing) (Firat et al., 2016b; Johnson et al., 2017; Al- Shedivat and Parikh, 2019; Gu et al., 2019). Despite these potential benefits, multilingual NMT tends to underperform its bilingual coun- terparts (Johnson et al., …

Content truncated to 500 words