Improving Multilingual Translation by Representation and Gradient Regularization

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7266–7279 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 7266 Improving Multilingual Translation by Representation and Gradient Regularization Yilin Yang1,∗, Akiko Eriguchi2, Alexandre Muzio2, Prasad Tadepalli1, Stefan Lee1, Hany Hassan2 1Oregon State University 2Microsoft 1{yangyil,tadepall,leestef}@oregonstate.edu 2{akikoe,alferre,hanyh}@microsoft.com Abstract Multilingual Neural Machine Translation (NMT) enables one model to serve all trans- lation directions, including ones that are un- seen during training, i.e. zero-shot translation. Despite being theoretically attractive, current models often produce low quality translations – commonly failing to even produce outputs in the right target language. In this work, we observe that off-target translation is dominant even in strong multilingual systems, trained on massive multilingual corpora. To address this issue, we propose a joint approach to regular- ize NMT models at both representation-level and gradient-level. At the representation level, we leverage an auxiliary target language pre- diction task to regularize decoder outputs to re- tain information about the target language. At the gradient level, we leverage a small amount of direct data (in thousands of sentence pairs) to regularize model gradients. Our results demonstrate that our approach is highly effec- tive in both reducing off-target translation oc- currences and improving zero-shot translation performance by +5.59 and +10.38 BLEU on WMT and OPUS datasets respectively. More- over, experiments show that our method also works well when the small amount of direct data is not available.1 1 Introduction With Neural Machine Translation becoming the state-of-the-art approach in bilingual machine trans- lations (Bahdanau et al., 2015; Vaswani et al., 2017), Multilingual NMT systems have increas- ingly gained attention due to their deployment ef- ficiency. One conceptually attractive advantage of Multilingual NMT (Johnson et al., 2017) is its ca- pability to translate between multiple source and target languages with only one model, where many ∗Work done while interning at Microsoft. 1Codes and rebuilt OPUS data at: https://github. com/yilinyang7/fairseq_multi_fix WMT Fr-De De-Fr Cs-De De-Cs Baseline 51.60% 39.80% 13.10% 20.50% OPUS Fr-De De-Fr Fr-Ru Ru-Fr Baseline 95.15% 93.70% 68.85% 91.20% Table 1: Off-target translation percentages on WMT and OPUS Testsets. directions2 are trained in a zero-shot manner. Despite its theoretical benefits, Multilingual NMT often suffers from target language interfer- ence (Johnson et al., 2017; Wang et al., 2020c). Specifically, Johnson et al. (2017) found that Mul- tilingual NMT often improves performance com- pared to bilingual models in many-to-one setting (translating other languages into English), yet often hurts performance in one-to-many setting (translat- ing English into other languages). Several other works (Wang et al., 2018; Arivazhagan et al., 2019; Tang et al., 2020) also confirm one-to-many trans- lation to be more challenging than many-to-one. Another widely observed phenomenon is that the current multilingual system on zero-shot transla- tions faces serious off-target translation issue (Gu et al., 2019; Zhang et al., 2020) where the gener- ated target text is not in the intended language. For example, Table 1 shows the percentage of off-target translations appearing between high-resource lan- guages. These issues exemplify the internal failure of …

Content truncated to 500 words