Using Monolingual Data in Neural Machine Translation: a Systematic Study

Proceedings of the Third Conference on Machine Translation (WMT), Volume 1: Research Papers, pages 144–155 Belgium, Brussels, October 31 - Novermber 1, 2018. c⃝2018 Association for Computational Linguistics https://doi.org/10.18653/v1/W18-64015 Using Monolingual Data in Neural Machine Translation: a Systematic Study Franck Burlot Lingua Custodia 1, Place Charles de Gaulle 78180 Montigny-le-Bretonneux franck.burlot@linguacustodia.com Franc¸ois Yvon LIMSI, CNRS, Universit´e Paris Saclay Campus Universitaire d’Orsay F-91 403 Orsay C´edex francois.yvon@limsi.fr Abstract Neural Machine Translation (MT) has radi- cally changed the way systems are developed. A major difference with the previous gener- ation (Phrase-Based MT) is the way mono- lingual target data, which often abounds, is used in these two paradigms. While Phrase- Based MT can seamlessly integrate very large language models trained on billions of sen- tences, the best option for Neural MT devel- opers seems to be the generation of artificial parallel data through back-translation - a tech- nique that fails to fully take advantage of exist- ing datasets. In this paper, we conduct a sys- tematic study of back-translation, comparing alternative uses of monolingual data, as well as multiple data generation procedures. Our findings confirm that back-translation is very effective and give new explanations as to why this is the case. We also introduce new data simulation techniques that are almost as effec- tive, yet much cheaper to implement. 1 Introduction The new generation of Neural Machine Transla- tion (NMT) systems is known to be extremely data hungry (Koehn and Knowles, 2017). Yet, most ex- isting NMT training pipelines fail to fully take ad- vantage of the very large volume of monolingual source and/or parallel data that is often available. Making a better use of data is particularly criti- cal in domain adaptation scenarios, where paral- lel adaptation data is usually assumed to be small in comparison to out-of-domain parallel data, or to in-domain monolingual texts. This situation sharply contrasts with the previous generation of statistical MT engines (Koehn, 2010), which could seamlessly integrate very large amounts of non- parallel documents, usually with a large positive effect on translation quality. Such observations have been made repeatedly and have led to many innovative techniques to in- tegrate monolingual data in NMT, that we review shortly. The most successful approach to date is the proposal of Sennrich et al. (2016a), who use monolingual target texts to generate artificial parallel data via backward translation (BT). This technique has since proven effective in many sub- sequent studies. It is however very computation- ally costly, typically requiring to translate large sets of data. Determining the “right” amount (and quality) of BT data is another open issue, but we observe that experiments reported in the literature only use a subset of the available monolingual re- sources. This suggests that standard recipes for BT might be sub-optimal. This paper aims to better understand the strengths and weaknesses of BT and to design more principled techniques to improve its effects. More specifically, we seek to answer the following questions: since there are many ways to generate pseudo parallel corpora, how …

Content truncated to 500 words