XLM-D: Decorate Cross-lingual Pre-training Model as Non-Autoregressive Neural Machine Translation

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6934 - 6946 December 7-11, 2022 ©2022 Association for Computational Linguistics XLM-D: Decorate Cross-lingual Pre-training Model as Non-Autoregressive Neural Machine Translation Yong Wang1∗ Shilin He2 Guanhua Chen3† Yun Chen4 Daxin Jiang2† 1Tencent Corporation 2Microsoft Corporation 3Southern University of Science and Technology 4Shanghai University of Finance and Economics seaywang@gmail.com shilin.he@microsoft.com ghchen08@gmail.com yunchen@sufe.edu.cn djiang@microsoft.com Abstract Pre-training language models have achieved thriving success in numerous natural language understanding and autoregressive generation tasks, but non-autoregressive generation in applications such as machine translation has not sufficiently benefited from the pre-training paradigm. In this work, we establish the connection between a pre-trained masked lan- guage model (MLM) and non-autoregressive generation on machine translation. From this perspective, we present XLM-D, which seam- lessly transforms an off-the-shelf cross-lingual pre-training model into a non-autoregressive translation (NAT) model with a lightweight yet effective decorator. Specifically, the dec- orator ensures the representation consistency of the pre-trained model and brings only one additional trainable parameter. Extensive ex- periments on typical translation datasets show that our models obtain state-of-the-art perfor- mance while realizing the inference speed- up by 19.9×. One striking result is that on WMT14 En⇒De, our XLM-D obtains 29.80 BLEU points with multiple iterations, which outperforms the previous mask-predict model by 2.77 points. 1 Introduction As a dominant pre-training paradigm in natural lan- guage processing (NLP), masked language models such as BERT and its variants (Devlin et al., 2019; Liu et al., 2019), were initially proposed and have achieved state-of-the-art performance on various natural language understanding tasks. For gener- ation tasks, previous studies either leverage the pre-trained BERT as an external component for representation fusing (Zhu et al., 2019) or simply initialize the generation models with a pre-trained model (Ma et al., 2020). Although straightforward, these methods suffer from either heavy compu- tation costs during inference or only supporting ∗Work done while at Microsoft Corporation. †Corresponding authors. autoregressive generation with sequential depen- dency. Additionally, these approaches result in inconsistency from two perspectives: 1) mismatch of architectures between pre-trained models and the generation model; 2) disagreement of training objectives between pre-training and autoregressive generation tasks. Different from previous studies, we reexamine the problem from another point of view. We find that the training objective in pre-trained MLM can align well with the one in non-autoregressive gen- eration, which is an emergent generation paradigm due to its excellent inference speed-up (Gu et al., 2018; Lee et al., 2018). Intuitively, both train- ing objectives are formalized as a series of inde- pendent token predictions in the output sequence. Based on this observation, we incorporate a pre- trained MLM into the non-autoregressive genera- tion. Specifically, in this work, we focus on the classic non-autoregressive machine translation task. Albeit studied for ages, NAT models have not per- formed very well, lagging behind the autoregres- sive translation counterpart. We propose to adapt the cross-lingual pre- training model (XLMR) into NAT models with a lightweight, effective and user-configurable dec- orator. Following Occam’s razor principle and …

Content truncated to 500 words