Alternated Training with Synthetic and Authentic Data for Neural Machine Translation

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1828–1834 August 1–6, 2021. ©2021 Association for Computational Linguistics 1828 Alternated Training with Synthetic and Authentic Data for Neural Machine Translation Rui Jiao1,3,4, Zonghan Yang1,3,4, Maosong Sun1,3,4,5, and Yang Liu∗ 1,2,3,4,5 1Department of Computer Science and Technology, Tsinghua University, Beijing, China 2Institute for AI Industry Research, Tsinghua University, Beijing, China 3Institute for Artificial Intelligence, Tsinghua University, Beijing, China 4Beijing National Research Center for Information Science and Technology 5Beijing Academy of Artificial Intelligence Abstract While synthetic bilingual corpora have demon- strated their effectiveness in low-resource neu- ral machine translation (NMT), adding more synthetic data often deteriorates translation performance. In this work, we propose al- ternated training with synthetic and authen- tic data for NMT. The basic idea is to al- ternate synthetic and authentic corpora itera- tively during training. Compared with previ- ous work, we introduce authentic data as guid- ance to prevent the training of NMT models from being disturbed by noisy synthetic data. Experiments on Chinese-English and German- English translation tasks show that our ap- proach improves the performance over several strong baselines. We visualize the BLEU land- scape to further investigate the role of authen- tic and synthetic data during alternated train- ing. From the visualization, we find that au- thentic data helps to direct the NMT model parameters towards points with higher BLEU scores and leads to consistent translation per- formance improvement. 1 Introduction While recent years have witnessed the rapid de- velopment of Neural Machine Translation (NMT) (Sutskever et al., 2014; Bahdanau et al., 2015; Gehring et al., 2017; Vaswani et al., 2017), it heav- ily relies on large-scale, high-quality bilingual cor- pora. Due to the expense and scarcity of authentic corpora, synthetic data has played a significant role in boosting translation quality (He et al., 2016; Sen- nrich et al., 2016a; Zhang and Zong, 2016; Cheng et al., 2016; Fadaee et al., 2017). Existing approaches to synthesizing data in NMT focus on leveraging monolingual data in the training process. Among them, back-translation (BT) (Sennrich et al., 2016a) has been widely used ∗ Corresponding author: Yang Liu to generate synthetic bilingual corpora by using a trained target-to-source NMT model to translate large-scale target-side monolingual corpora. Such synthetic data can be used to improve source-to- target NMT models. Despite the effectiveness of back-translation, the synthetic data inevitably con- tains noise and erroneous translations. As a mat- ter of fact, it has been widely observed that while BT is capable of benefiting NMT models by us- ing relatively small-scale synthetic data, further increasing the quantity often deteriorates transla- tion performance (Edunov et al., 2018; Wu et al., 2019; Caswell et al., 2019). This problem has attracted increasing attention in the NMT community (Edunov et al., 2018; Wang et al., 2019). One direction to alleviate the prob- lem is to add noise or a special tag on the source side of synthetic data, which enables NMT mod- els to distinguish between authentic and synthetic data (Edunov et al., 2018; …

Content truncated to 500 words