Causes and Cures for Interference in Multilingual Translation

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 15849–15863 July 9-14, 2023 ©2023 Association for Computational Linguistics Causes and Cures for Interference in Multilingual Translation Uri Shahamτ Maha Elbayadµ Vedanuj Goswamiµ Omer Levyτµ Shruti Bhosaleµ τ The Blavatnik School of Computer Science, Tel Aviv University µ Meta AI Abstract Multilingual machine translation models can benefit from synergy between different lan- guage pairs, but also suffer from interference. While there is a growing number of sophisti- cated methods that aim to eliminate interfer- ence, our understanding of interference as a phenomenon is still limited. This work iden- tifies the main factors that contribute to in- terference in multilingual machine translation. Through systematic experimentation, we find that interference (or synergy) are primarily de- termined by model size, data size, and the pro- portion of each language pair within the total dataset. We observe that substantial interfer- ence occurs mainly when the model is very small with respect to the available training data, and that using standard transformer configu- rations with less than one billion parameters largely alleviates interference and promotes synergy. Moreover, we show that tuning the sampling temperature to control the proportion of each language pair in the data is key to bal- ancing the amount of interference between low and high resource language pairs effectively, and can lead to superior performance overall. 1 Introduction Multilingual machine translation models can bene- fit from transfer between different language pairs (synergy), but may also suffer from interference (Ha et al., 2016; Firat et al., 2016; Aharoni et al., 2019; Arivazhagan et al., 2019). While there are methods to reduce interference and achieve better performance (Wang et al., 2020a; Kreutzer et al., 2021; Wang et al., 2021), such approaches are often compute intensive, and do not always work (Xin et al., 2022). In this work, we demonstrate that interference in multilingual translation largely oc- curs when the model is very small compared to the abundance of training data, and that the simple principled approach of enlarging the model and tuning the data sampling temperature provides a consistent solution to the interference problem that can even promote synergy. This work methodically deduces the most sim- ple ways of reducing interference in multilingual translation. We begin by inquiring what are the dominant factors that may interfere with learning to translate a particular language pair of focus s → t, in the context of learning a multilingual translation model with many different language pairs. Con- trolled experiments show that besides model size and number of s → t training examples, the main factor that correlates with the level of interference is the proportion of focus pair examples (s → t) observed out of the total number of examples (all language pairs) seen at each training step on aver- age. Surprisingly, aspects like language similarity or number of translation directions have a much smaller effect. In model and data scaling experiments, we ob- serve that interference mainly occurs in …

Content truncated to 500 words