Breaking Down Multilingual Machine Translation

Findings of the Association for Computational Linguistics: ACL 2022, pages 2766 - 2780 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Breaking Down Multilingual Machine Translation Ting-Rui Chiang1 Yi-Pei Chen2 Yi-Ting Yeh1 Graham Neubig1 1Carnegie Mellon University, 2The University of Tokyo {tingruic,yitingye,gneubig}@cs.cmu.edu ypc@g.ecc.u-tokyo.ac.jp Abstract While multilingual training is now an essen- tial ingredient in machine translation (MT) sys- tems, recent work has demonstrated that it has different effects in different multilingual set- tings, such as many-to-one, one-to-many, and many-to-many learning. These training set- tings expose the encoder and the decoder in a machine translation model with different data distributions. In this paper, we examine how different varieties of multilingual training con- tribute to learning these two components of the MT model. Specifically, we compare bilingual models with encoders and/or decoders initial- ized by multilingual training. We show that multilingual training is beneficial to encoders in general, while it only benefits decoders for low-resource languages (LRLs). We further find the important attention heads for each lan- guage pair and compare their correlations dur- ing inference. Our analysis sheds light on how multilingual translation models work and en- ables us to propose methods to improve per- formance by training with highly related lan- guages. Our many-to-one models for high- resource languages and one-to-many models for LRL outperform the best results reported by Aharoni et al. (2019). 1 Introduction Multilingual training regimens (Dong et al., 2015; Firat et al., 2016; Ha et al., 2016) are now a key ele- ment of natural language processing, especially for low-resource languages (LRLs) (Neubig and Hu, 2018; Aharoni et al., 2019). These algorithms are presumed to be helpful because they leverage syn- tactic or semantic similarities between languages, and transfer processing abilities across language boundaries. In general, English is used as a central language due to its data availability, and three different multi- lingual training settings are considered: (1) one-to- many: training a model with languages pairs from English to many other languages. (2) many-to-one: training a model with languages pairs from many languages to English (3) many-to-many: training a model with the union of the above two settings’ data. (1) and (3) can be used for English to other (En-X) translation, while (2) and (3) can be used for other to English (X-En) translation. However, multilingual training has not proven equally helpful in every setting. Arivazhagan et al. (2019) showed that many-to-one training improves performance over bilingual baselines more than one-to-many does. In this paper, we consider this result from the point of view of the components of the MT model. In the many-to-one setting, the model’s inputs are from different language distri- butions so the encoder can be considered a multi- domain model, whereas the decoder is trained on a single distribution. In the one-to-many setting, it is the opposite: the encoder shares data, and the decoder is multi-domain. While there are recent studies analyzing multilingual translation models (Kudugunta et al., 2019; Voita et al., 2019a; Aji et al., 2020; Mueller et al., 2020), in …

Content truncated to 500 words