Encouraging Lexical Translation Consistency for Document-Level Neural Machine Translation

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 15–26 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 15 Zero-Shot Cross-Lingual Transfer of Neural Machine Translation with Multilingual Pretrained Encoders Guanhua Chen1∗, Shuming Ma2, Yun Chen3†, Li Dong2 Dongdong Zhang2, Jia Pan1, Wenping Wang4,1, Furu Wei2 1The University of Hong Kong; 2Microsoft Research 3Shanghai University of Finance and Economics; 4Texas A&M University {ghchen,jpan,wenping}@cs.hku.hk, yunchen@sufe.edu.cn, {shumma, lidong1, dozhang, fuwei}@microsoft.com Abstract Previous work mainly focuses on improving cross-lingual transfer for NLU tasks with a multilingual pretrained encoder (MPE), or im- proving the performance on supervised ma- chine translation with BERT. However, it is under-explored that whether the MPE can help to facilitate the cross-lingual transferability of NMT model. In this paper, we focus on a zero-shot cross-lingual transfer task in NMT. In this task, the NMT model is trained with parallel dataset of only one language pair and an off-the-shelf MPE, then it is directly tested on zero-shot language pairs. We pro- pose SixT, a simple yet effective model for this task. SixT leverages the MPE with a two-stage training schedule and gets further improvement with a position disentangled en- coder and a capacity-enhanced decoder. Using this method, SixT significantly outperforms mBART, a pretrained multilingual encoder- decoder model explicitly designed for NMT, with an average improvement of 7.1 BLEU on zero-shot any-to-English test sets across 14 source languages. Furthermore, with much less training computation cost and training data, our model achieves better performance on 15 any-to-English test sets than CRISS and m2m-100, two strong multilingual NMT base- lines. 1 Introduction Multilingual pretrained encoders (MPE) such as mBERT (Wu and Dredze, 2019), XLM (Con- neau and Lample, 2019), and XLM-R (Conneau et al., 2020) have shown remarkably strong re- sults on zero-shot cross-lingual transfer mainly for natural language understanding (NLU) tasks, including named entity recognition (NER), ques- tion answering (QA) and natural language infer- ence (NLI). These methods jointly train a Trans- former (Vaswani et al., 2017) encoder to perform ∗ Contribution during internship at Microsoft Research. † Corresponding author. Encoder body Decoder body Encoder embed Decoder embed (2) Training in the second stage De En Multilingual pretrained model Zero-shot Translate Ru En Zh En Hi En Fi En Encoder body Decoder body Encoder embed Decoder embed (1) Training in the first stage De En Multilingual Pretrained Encoder Zero-shot Translate En NMT model Fi Hi Zh Train Figure 1: In the zero-shot cross-lingual NMT trans- fer task, the model is trained with parallel dataset of only one language pair (such as De-En) and a multilin- gual pretrained encoder. The trained model is tested on many-to-one language pairs (like Fi/Hi/Zh-En) in a zero-shot manner. Monolingual text of the to-be-tested source languages is not available in this task. masked language modeling task in multiple lan- guages. The pretrained model is then fine-tuned on a downstream NLU task using labeled data in a single language and evaluated on the same task in other languages. With this pretraining and fine- tuning approach, the …

Content truncated to 500 words