Improving Neural Machine Translation Models with Monolingual Data

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 86–96, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics Improving Neural Machine Translation Models with Monolingual Data Rico Sennrich and Barry Haddow and Alexandra Birch School of Informatics, University of Edinburgh {rico.sennrich,a.birch}@ed.ac.uk, bhaddow@inf.ed.ac.uk Abstract Neural Machine Translation (NMT) has obtained state-of-the art performance for several language pairs, while only us- ing parallel data for training. Target- side monolingual data plays an impor- tant role in boosting fluency for phrase- based statistical machine translation, and we investigate the use of monolingual data for NMT. In contrast to previous work, which combines NMT models with sep- arately trained language models, we note that encoder-decoder NMT architectures already have the capacity to learn the same information as a language model, and we explore strategies to train with monolin- gual data without changing the neural net- work architecture. By pairing monolin- gual training data with an automatic back- translation, we can treat it as additional parallel training data, and we obtain sub- stantial improvements on the WMT 15 task English↔German (+2.8–3.7 BLEU), and for the low-resourced IWSLT 14 task Turkish→English (+2.1–3.4 BLEU), ob- taining new state-of-the-art results. We also show that fine-tuning on in-domain monolingual and parallel data gives sub- stantial improvements for the IWSLT 15 task English→German. 1 Introduction Neural Machine Translation (NMT) has obtained state-of-the art performance for several language pairs, while only using parallel data for training. Target-side monolingual data plays an important role in boosting fluency for phrase-based statisti- The research presented in this publication was conducted in cooperation with Samsung Electronics Polska sp. z o.o. - Samsung R&D Institute Poland. cal machine translation, and we investigate the use of monolingual data for NMT. Language models trained on monolingual data have played a central role in statistical machine translation since the first IBM models (Brown et al., 1990). There are two major reasons for their importance. Firstly, word-based and phrase-based translation models make strong independence as- sumptions, with the probability of translation units estimated independently from context, and lan- guage models, by making different independence assumptions, can model how well these translation units fit together. Secondly, the amount of avail- able monolingual data in the target language typi- cally far exceeds the amount of parallel data, and models typically improve when trained on more data, or data more similar to the translation task. In (attentional) encoder-decoder architectures for neural machine translation (Sutskever et al., 2014; Bahdanau et al., 2015), the decoder is es- sentially an RNN language model that is also con- ditioned on source context, so the first rationale, adding a language model to compensate for the in- dependence assumptions of the translation model, does not apply. However, the data argument is still valid in NMT, and we expect monolingual data to be especially helpful if parallel data is sparse, or a poor fit for the translation task, for instance be- cause of a domain mismatch. In contrast to previous work, which integrates a …

Content truncated to 500 words