Unifying the Convergences in Multilingual Neural Machine Translation

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6822 - 6835 December 7-11, 2022 ©2022 Association for Computational Linguistics Unifying the Convergences in Multilingual Neural Machine Translation Yichong Huang Xiaocheng Feng Xinwei Geng Bing Qin Harbin Institute of Technology, China {ychuang, xcfeng, xwgeng, qinb}@ir.hit.edu.cn Abstract Although all-in-one-model multilingual neu- ral machine translation (MNMT) has achieved remarkable progress, the convergence in- consistency in the joint training is ignored, i.e.,different language pairs reaching conver- gence in different epochs. This leads to the trained MNMT model over-fitting low-resource language translations while under-fitting high- resource ones. In this paper, we propose a novel training strategy named LSSD (Language- Specific Self-Distillation), which can allevi- ate the convergence inconsistency and help MNMT models achieve the best performance on each language pair simultaneously. Specif- ically, LSSD picks up language-specific best checkpoints for each language pair to teach the current model on the fly. Furthermore, we sys- tematically explore three sample-level manipu- lations of knowledge transferring. Experimen- tal results on three datasets show that LSSD obtains consistent improvements towards all language pairs and achieves the state-of-the-art 1 . 1 Introduction Neural machine translation (NMT) (Kalchbrenner and Blunsom, 2013; Sutskever et al., 2014) has witnessed enormous and significant progress, in- cluding network structures (Cho et al., 2014; Bah- danau et al., 2015; Gehring et al., 2017; Vaswani et al., 2017), attention mechanism (Luong et al., 2015; Liu et al., 2016; Shen et al., 2018) and decod- ing strategies (Xia et al., 2017; Geng et al., 2018; Zhou et al., 2019). While achieving promising per- formance (Wu et al., 2016; Hassan et al., 2018), widely-used bilingual NMT actually causes huge computational cost especially when tackling numer- ous language pairs, thereby facilitating the recent emergence of multilingual NMT (Ha et al., 2016; Firat et al., 2016; Johnson et al., 2017; Lu et al., 1Our code is publicly available at https://github. com/OrangeInSouth/LSSD. 4.05 4.15 4.25 4.35 4.45 10 15 20 25 30 35 40 (a) bos→eng 14 24 4.70 4.81 4.93 5.04 5.15 10 15 20 25 30 35 40 (b) mar→eng 17 24 3.52 3.54 3.55 3.57 3.58 20 30 40 50 60 (c) fra→eng 24 49 4.34 4.36 4.38 4.40 4.42 20 30 40 50 60 (d) kor→eng 49 24 Figure 1: Loss curves of a multilingual NMT model trained to translate 4 languages into English. The upper two boxes illustrate low-resource directions. And the high-resource ones are placed in the bottom part. The x-axis and y-axis indicate training epochs and dev losses respectively. Dotted lines mark language-specific best checkpoints (colorful ones) as well as the overall best checkpoint (the black one). 2018; Aharoni et al., 2019). By directly translating multiple language pairs with one model, multilin- gual NMT (Tan et al., 2019; Zhang et al., 2020; Fan et al., 2021; Zhang et al., 2021a) quadratically accelerates deployment and effectively encourages transfer learning between similar languages, which greatly benefits low-resource directions (Arivazha- gan et al., 2019) and successfully enables zero-shot translation (Gu et al., 2019). …

Content truncated to 500 words