Multilingual Mix: Example Interpolation Improves Multilingual Neural Machine Translation

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 4092 - 4102 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Multilingual Mix: Example Interpolation Improves Multilingual Neural Machine Translation Yong Cheng, Ankur Bapna, Orhan Firat, Yuan Cao, Pidong Wang, and Wolfgang Macherey Google Research {chengyong,ankurbpn,orhanf,yuancao,pidong,wmach}@google.com Abstract Multilingual neural machine translation mod- els are trained to maximize the likelihood of a mix of examples drawn from multiple language pairs. The dominant inductive bias applied to these models is a shared vocab- ulary and a shared set of parameters across languages; the inputs and labels correspond- ing to examples drawn from different lan- guage pairs might still reside in distinct sub- spaces. In this paper, we introduce multilin- gual crossover encoder-decoder (mXEncDec) to fuse language pairs at an instance level. Our approach interpolates instances from different language pairs into joint ‘crossover examples’ in order to encourage sharing input and output spaces across languages. To ensure better fu- sion of examples in multilingual settings, we propose several techniques to improve exam- ple interpolation across dissimilar languages under heavy data imbalance. Experiments on a large-scale WMT multilingual dataset demon- strate that our approach significantly improves quality on English-to-Many, Many-to-English and zero-shot translation tasks (from +0.5 BLEU up to +5.5 BLEU points). Results on code-switching sets demonstrate the capability of our approach to improve model generaliza- tion to out-of-distribution multilingual exam- ples. We also conduct qualitative and quantita- tive representation comparisons to analyze the advantages of our approach at the representa- tion level. 1 Introduction Multilingual modeling has been receiving increas- ing research attention over the past few years, aris- ing from successful demonstrations of improved quality across a variety of tasks, languages and modalities (Lample and Conneau, 2019; Arivazha- gan et al., 2019b; Conneau et al., 2021). The suc- cess of these models is typically ascribed to vocabu- lary sharing, parameter tying and implicit pivoting through dominant languages like English (Conneau et al., 2020). These conventional techniques are effective, but might not be exploiting the full poten- tial of multilingual models to learn the underlying inductive bias: the learning signal from one lan- guage should benefit the quality of other languages (Caruana, 1997; Arivazhagan et al., 2019b). Here we study two related issues that exist in the context of multilingual Neural Machine Transla- tion (NMT) training (Dong et al., 2015; Firat et al., 2016a; Johnson et al., 2017). First, NMT mod- els (Bahdanau et al., 2015; Vaswani et al., 2017) are trained with maximum likelihood estimation which has a strong tendency to overfit and even memo- rize observed training examples, particularly pos- ing challenges for low resource languages (Zhang et al., 2018). Second, training examples from dis- tinct language pairs are separately fed into multi- lingual NMT models without any explicit instance- level sharing (with the exception of multi-source NMT (Zoph and Knight, 2016; Firat et al., 2016b)); as a consequence, given large enough capacity, the models have the liberty to map representations of different languages …

Content truncated to 500 words