Exploiting Monolingual Data at Scale for Neural Machine Translation

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 4207–4216, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 4207 Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 4198–4207, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 4198 Exploiting Monolingual Data at Scale for Neural Machine Translation Lijun Wu1,⇤, Yiren Wang2,⇤, Yingce Xia3,†, Tao Qin3, Jianhuang Lai1, Tie-Yan Liu3 1School of Data and Computer Science, Sun Yat-sen University; 2University of Illinois at Urbana-Champaign; 3Microsoft Research Asia; 1{wulijun3@mail2, stsljh@mail}.sysu.edu.cn; 2yiren@illinois.edu; 3{Yingce.Xia, taoqin, tyliu}@microsoft.com Abstract While target-side monolingual data has been proven to be very useful to improve neural ma- chine translation (briefly, NMT) through back translation, source-side monolingual data is not well investigated. In this work, we study how to use both the source-side and target- side monolingual data for NMT, and propose an effective strategy leveraging both of them. First, we generate synthetic bitext by translat- ing monolingual data from the two domains into the other domain using the models pre- trained on genuine bitext. Next, a model is trained on a noised version of the concate- nated synthetic bitext where each source se- quence is randomly corrupted. Finally, the model is fine-tuned on the genuine bitext and a clean version of a subset of the synthetic bi- text without adding any noise. Our approach achieves state-of-the-art results on WMT16, WMT17, WMT18 English$German transla- tions and WMT19 German!French transla- tions, which demonstrate the effectiveness of our method. We also conduct a comprehensive study on how each part in the pipeline works. 1 Introduction Neural machine translation (briefly, NMT) (Bah- danau et al., 2014; Gehring et al., 2017; Wu et al., 2016; Vaswani et al., 2017) is well-known for it- s outstanding performance (Hassan et al., 2018), which usually relies on large-scale bitext for train- ing. However, high quality bitext is always limit- ed and costly to collect. In contrast, there exists large amount of monolingual data which can be leveraged to augment the training corpus. How to effectively leverage monolingual data is an impor- tant research topic for NMT and there are plenty of works studying this problem (Gulcehre et al., ⇤The first two authors contributed equally to this work. This work was conducted at Microsoft Research Asia. †Corresponding author. 2015, 2017; Sennrich et al., 2016a; He et al., 2016; Zhang and Zong, 2016; Zhang et al., 2018; Cheng et al., 2016; Wang et al., 2018; Chinea-Rios et al., 2017; Wu et al., 2018). Among them, one of the most cited approach is back translation (briefly, BT) (Sennrich et al., 2016a), which leverages the target-side monolin- gual data. Specifically, a target-to-source transla- tion model (trained on the genuine bitext) is used to translate target-side monolingual sentences in- to the source domain to generate a set of synthetic bitext, which is then used together with the gen- …

Content truncated to 500 words