Universal Conditional Masked Language Pre-training for Neural Machine Translation

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 6379 - 6391 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Universal Conditional Masked Language Pre-training for Neural Machine Translation Pengfei Li1 Liangyou Li1 Meng Zhang1 Minghao Wu2 Qun Liu1 1Huawei Noah’s Ark Lab 2Monash University {lipengfei111, liliangyou, zhangmeng92, qun.liu}@huawei.com minghao.wu@monash.edu Abstract Pre-trained sequence-to-sequence models have significantly improved Neural Machine Translation (NMT). Different from prior works where pre-trained models usually adopt an unidirectional decoder, this paper demonstrates that pre-training a sequence- to-sequence model but with a bidirectional decoder can produce notable performance gains for both Autoregressive and Non- autoregressive NMT. Specifically, we propose CeMAT, a conditional masked language model pre-trained on large-scale bilingual and monolingual corpora in many languages.1 We also introduce two simple but effective methods to enhance the CeMAT, aligned code-switching & masking and dynamic dual-masking. We conduct extensive experi- ments and show that our CeMAT can achieve significant performance improvement for all scenarios from low- to extremely high- resource languages, i.e., up to +14.4 BLEU on low-resource and +7.9 BLEU on average for Autoregressive NMT. For Non-autoregressive NMT, we demonstrate it can also produce consistent performance gains, i.e., up to +5.3 BLEU. To the best of our knowledge, this is the first work to pre-train a unified model for fine-tuning on both NMT tasks. 1 Introduction Pre-trained language models have been widely adopted in NLP tasks (Devlin et al., 2019; Radford and Narasimhan, 2018). For example, XLM (Con- neau and Lample, 2019) demonstrated that cross- lingual pre-training is effective in improving neu- ral machine translation (NMT), especially on low- resource languages. These methods all directly pre- train a bidirectional encoder or an unidirectional decoder. The encoder and decoder in NMT models are then independently initialized with them and 1Code, data, and pre-trained models are avail- able at https://github.com/huawei-noah/ Pretrained-Language-Model/CeMAT Approach Enc. Dec. Mono. Para. mBERT (Devlin et al., 2019) • • XLM (Conneau and Lample, 2019) • • • MASS (Song et al., 2019) • → • mBART (Liu et al., 2020) • → • mRASP (Lin et al., 2020) • → • CeMAT (Ours) • ⇐⇒ • • Table 1: Comparison and summary of existing pre- trained models for machine translation. Enc: encoder; Dec: decoder; Mono: monolingual; Para: bilingual. “•” denotes the corresponding model is pre-trained or the corresponding data is used. “→” denotes the de- coder of model is unidirectional, “⇐⇒” denotes the de- coder is bidirectional. fine-tuned (Guo et al., 2020; Zhu et al., 2020). Re- cently, pre-training standard sequence-to-sequence (Seq2Seq) models has shown significant improve- ments and become a popular paradigm for NMT tasks (Song et al., 2019; Liu et al., 2020; Lin et al., 2020). However, some experimental results from XLM (Conneau and Lample, 2019) have shown that the decoder module initialized by the pre-trained bidi- rectional masked language model (MLM) (Devlin et al., 2019), rather than the unidirectional causal language model (CLM, Radford and Narasimhan, 2018), would achieve better results on Autore- gressive NMT …

Content truncated to 500 words