Understanding and Improving Sequence-to-Sequence Pretraining for Neural Machine Translation

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 2591 - 2600 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Understanding and Improving Sequence-to-Sequence Pretraining for Neural Machine Translation Wenxuan Wang1⇤ Wenxiang Jiao2 Yongchang Hao3 Xing Wang2 Shuming Shi2 Zhaopeng Tu2† Michael R. Lyu1 1Department of Computer Science and Engineering, The Chinese University of Hong Kong {wxwang,lyu}@cse.cuhk.edu.hk 2Tencent AI Lab {joelwxjiao,brightxwang,shumingshi,zptu}@tencent.com 3University of Alberta yongcha1@ualberta.ca Abstract In this paper, we present a substantial step in better understanding the SOTA sequence-to- sequence (Seq2Seq) pretraining for neural ma- chine translation (NMT). We focus on study- ing the impact of the jointly pretrained de- coder, which is the main difference between Seq2Seq pretraining and previous encoder- based pretraining approaches for NMT. By carefully designing experiments on three lan- guage pairs, we find that Seq2Seq pretraining is a double-edged sword: On one hand, it helps NMT models to produce more diverse transla- tions and reduce adequacy-related translation errors. On the other hand, the discrepancies be- tween Seq2Seq pretraining and NMT finetun- ing limit the translation quality (i.e., domain discrepancy) and induce the over-estimation issue (i.e., objective discrepancy). Based on these observations, we further propose simple and effective strategies, named in-domain pre- training and input adaptation to remedy the do- main and objective discrepancies, respectively. Experimental results on several language pairs show that our approach can consistently im- prove both translation performance and model robustness upon Seq2Seq pretraining. 1 Introduction There has been a wealth of research over the past several years on self-supervised pre-training for nat- ural language processing tasks (Devlin et al., 2019; Liu et al., 2019; Conneau et al., 2020; Jiao et al., 2020a), which aims at transferring the knowledge of large-scale unlabeled data to downstream tasks with labeled data. Despite its success in other un- derstanding and generation tasks, self-supervised pretraining is not a common practice in machine translation (MT). One possible reason is the ar- chitecture discrepancy between pretraining model ⇤ Work was mainly done when Wenxuan Wang and Yongchang Hao were interning at Tencent AI Lab. †Zhaopeng Tu is the corresponding author. (e.g., Transformer encoder) and NMT models (e.g., Transformer encoder-decoder). To remedy the architecture gap, several re- searchers propose sequence-to-sequence (Seq2Seq) pretraining models for machine translation, e.g., MASS (Song et al., 2019) and BART (Zhu et al., 2019; Lewis et al., 2020). Recently, Liu et al. (2020) extend BART by training on large-scale multilingual language data (i.e., mBART), lead- ing to significant improvement on translation per- formance across various language pairs. While previous pretraining approaches for NMT gener- ally focus only on Transformer encoder (Lample and Conneau, 2019), mBART pretrains a complete autoregressive Seq2Seq model by recovering the input sentences that are noised by masking phrases. One research question naturally arises: how much does the jointly pretrained decoder matter? In this work, we present a substantial step in bet- ter understanding the SOTA Seq2Seq pretraining model. We take a fine-grained look at the impact of the jointly pretrained decoder by …

Content truncated to 500 words