On the Copying Behaviors of Pre-Training for Neural Machine Translation

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4265–4275 August 1–6, 2021. ©2021 Association for Computational Linguistics 4265 On the Copying Behaviors of Pre-Training for Neural Machine Translation Xuebo Liu1∗, Longyue Wang2, Derek F. Wong1, Liang Ding3, Lidia S. Chao1, Shuming Shi2 and Zhaopeng Tu2 1NLP2CT Lab, Department of Computer and Information Science, University of Macau 2Tencent AI Lab 3The University of Sydney nlp2ct.xuebo@gmail.com, {derekfw,lidiasc}@um.edu.com {vinnylywang,shumingshi,zptu}@tencent.com ldin3097@sydney.edu.au Abstract Previous studies have shown that initializ- ing neural machine translation (NMT) mod- els with the pre-trained language models (LM) can speed up the model training and boost the model performance. In this work, we identify a critical side-effect of pre-training for NMT, which is due to the discrepancy between the training objectives of LM-based pre-training and NMT. Since the LM objec- tive learns to reconstruct a few source tokens and copy most of them, the pre-training ini- tialization would affect the copying behaviors of NMT models. We provide a quantitative analysis of copying behaviors by introducing a metric called copying ratio, which empirically shows that pre-training based NMT models have a larger copying ratio than the standard one. In response to this problem, we propose a simple and effective method named copy- ing penalty to control the copying behaviors in decoding. Extensive experiments on both in- domain and out-of-domain benchmarks show that the copying penalty method consistently improves translation performance by control- ling copying behaviors for pre-training based NMT models. Source code is freely avail- able at https://github.com/SunbowLiu/ CopyingPenalty. 1 Introduction Self-supervised pre-training (Devlin et al., 2019; Song et al., 2019), which acquires general knowl- edge from a large amount of unlabeled data to help better and faster learning downstream tasks, has an intuitive appeal for neural machine translation (NMT; Bahdanau et al., 2015; Vaswani et al., 2017). One direct way to utilize pre-trained knowledge is initializing the NMT model with a pre-trained language model (LM) before training it on paral- lel data (Conneau and Lample, 2019; Liu et al., ∗Work was done when Xuebo Liu and Liang Ding were interning at Tencent AI Lab. LM Pre-Training: LPT = − log P(x|�x) Source Military Field Marshal Hussein in attendance. Target Military ruler Field Marshal Hussein Tantawi was in attendance. NMT Training: LNMT = − log P(y|x) Source Military ruler Field Marshal Hussein Tantawi was in attendance. Target Der Milit¨arf¨uhrer Feldmarschall Hus- sein Tantawi war anwesend. Table 1: Training objective gap between Seq2Seq LM pre-training and NMT training. LM learns to recon- struct a few source tokens and copy most of them, while NMT learns more translation rather than copy- ing. Underlines denote artificial noises, and highlights indicate expected copying tokens. 2020). As a range of surface, syntactic and seman- tic information has been encoded in the initialized parameters (Jawahar et al., 2019; Goldberg, 2019), they are expected to bring benefits to NMT models and hence the translation quality. However, there is a discrepancy between the training objective of sequence-to-sequence LM pre- training and that of NMT training. As shown in …

Content truncated to 500 words