Pre-training Multilingual Neural Machine Translation by Leveraging Alignment Information

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 2649–2663, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 2649 Pre-training Multilingual Neural Machine Translation by Leveraging Alignment Information Zehui Lin†,‡∗, Xiao Pan†∗, Mingxuan Wang†, Xipeng Qiu‡, Jiangtao Feng†, Hao Zhou†, Lei Li† †ByteDance AI Lab {panxiao.94,wangmingxuan.89,zhouhao.nlp,fengjiangtao,lileilab}@bytedance.com ‡School of Computer Science, Fudan University, Shanghai, China {linzh18,xpqiu}@fudan.edu.cn Abstract We investigate the following question for ma- chine translation (MT): can we develop a single universal MT model to serve as the common seed and obtain derivative and im- proved models on arbitrary language pairs? We propose mRASP, an approach to pre-train a universal multilingual neural machine trans- lation model. Our key idea in mRASP is its novel technique of random aligned substitu- tion, which brings words and phrases with sim- ilar meanings across multiple languages closer in the representation space. We pre-train a mRASP model on 32 language pairs jointly with only public datasets. The model is then fine-tuned on downstream language pairs to obtain specialized MT models. We carry out extensive experiments on 42 translation direc- tions across a diverse settings, including low, medium, rich resource, and as well as trans- ferring to exotic language pairs. Experimental results demonstrate that mRASP achieves sig- nificant performance improvement compared to directly training on those target pairs. It is the first time to verify that multiple low- resource language pairs can be utilized to im- prove rich resource MT. Surprisingly, mRASP is even able to improve the translation quality on exotic languages that never occur in the pre- training corpus. Code, data, and pre-trained models are available at https://github. com/linzehui/mRASP. 1 Introduction Pre-trained language models such as BERT have been highly effective for NLP tasks (Peters et al., 2018; Devlin et al., 2019; Radford et al., 2019; Conneau and Lample, 2019; Liu et al., 2019; Yang et al., 2019). Pre-training and fine-tuning has been a successful paradigm. It is intriguing to discover a “BERT” equivalent – a pre-trained model – for ∗Equal contribution. The work was done when the first author was an intern at ByteDance. machine translation. In this paper, we study the following question: can we develop a single uni- versal MT model and derive specialized models by fine-tuning on an arbitrary pair of languages? While pre-training techniques are working very well for NLP task, there are still several limitations for machine translation tasks. First, pre-trained lan- guage models such as BERT are not easy to di- rectly fine-tune unless using some sophisticated techniques (Yang et al., 2020). Second, there is a discrepancy between existing pre-training objective and down-stream ones in MT. Existing pre-training approaches such as MASS (Song et al., 2019) and mBART (Liu et al., 2020) rely on auto-encoding objectives to pre-train the models, which are dif- ferent from translation. Therefore, their fine-tuned MT models still do not achieve adequate improve- ment. Third, existing MT pre-training approaches focus on using multilingual models to improve MT for low resource or medium resource languages. There has not been one …

Content truncated to 500 words