Alternative Input Signals Ease Transfer in Multilingual Machine Translation

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 5291 - 5305 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Alternative Input Signals Ease Transfer in Multilingual Machine Translation Simeng Sun1∗ Angela Fan2 James Cross2 Vishrav Chaudhary3† Chau Tran2 Philipp Koehn2 Francisco Guzmán2 University of Massachusetts Amherst1 Meta AI2 Microsoft Turing3 simengsun@umass.edu {angelafan,jcross,chau,pkoehn,fguzman}@fb.com vchaudhary@microsoft.com Abstract Recent work in multilingual machine transla- tion (MMT) has focused on the potential of positive transfer between languages, particu- larly cases where higher-resourced languages can benefit lower-resourced ones. While train- ing an MMT model, the supervision signals learned from one language pair can be trans- ferred to the other via the tokens shared by multiple source languages. However, the trans- fer is inhibited when the token overlap among source languages is small, which manifests naturally when languages use different writ- ing systems. In this paper, we tackle inhib- ited transfer by augmenting the training data with alternative signals that unify different writing systems, such as phonetic, romanized, and transliterated input. We test these signals on Indic and Turkic languages, two language families where the writing systems differ but languages still share common features. Our results indicate that a straightforward multi- source self-ensemble — training a model on a mixture of various signals and ensembling the outputs of the same model fed with different signals during inference — outperforms strong ensemble baselines by 1.3 BLEU on both lan- guage families. Further, we find that incor- porating alternative inputs via self-ensemble can be particularly effective in low-resource settings, leading to +5 BLEU when only 5% of the total training data is accessible. Fi- nally, our analysis demonstrates that includ- ing alternative signals yields more consistency and translates named entities more accurately, which is crucial for increased factuality of au- tomated systems. 1 Introduction Machine translation has seen great progress, with improvements in quality and successful commer- cial applications. However, the majority of this ∗Work done during an internship at Meta AI. †Work done while at Meta AI. improvement benefits languages with large quan- tities of high-quality training data (high-resource languages). Recently, researchers have focused on the development of multilingual translation mod- els (Aharoni et al., 2019; Fan et al., 2020) capa- ble of translating between many different language pairs rather than specialized models for each trans- lation direction. In particular, such multilingual models hold great promise for improving transla- tion quality for low-resource languages, as group- ing languages together allows them to benefit from linguistic similarities as well as shared data be- tween related languages. For example, training a translation system with combined Assamese and Bengali data would enable transfer learning be- tween the two languages. We investigate how to enable multilingual trans- lation models to optimally learn these similarities between languages and leverage this similarity to improve translation quality. The fundamental unit representing lingual similarity is the token — lan- guages that are similar often have similar words or phrases — and during training, translation mod- els can …

Content truncated to 500 words