Investigating Backtranslation in Neural Machine Translation

Investigating Backtranslation in Neural Machine Translation Alberto Poncelas, Dimitar Shterionov, Andy Way, Gideon Maillette de Buy Wenniger and Peyman Passban School of Computing, DCU, ADAPT Centre {firstname.lastname}@adaptcentre.ie Abstract A prerequisite for training corpus-based machine translation (MT) systems – ei- ther Statistical MT (SMT) or Neural MT (NMT) – is the availability of high-quality parallel data. This is arguably more impor- tant today than ever before, as NMT has been shown in many studies to outperform SMT, but mostly when large parallel cor- pora are available; in cases where data is limited, SMT can still outperform NMT. Recently researchers have shown that back-translating monolingual data can be used to create synthetic parallel corpora, which in turn can be used in combination with authentic parallel data to train a high- quality NMT system. Given that large collections of new parallel text become available only quite rarely, backtransla- tion has become the norm when building state-of-the-art NMT systems, especially in resource-poor scenarios. However, we assert that there are many un- known factors regarding the actual effects of back-translated data on the translation capabilities of an NMT model. Accord- ingly, in this work we investigate how us- ing back-translated data as a training cor- pus – both as a separate standalone dataset as well as combined with human-generated parallel data – affects the performance of an NMT model. We use incrementally larger amounts of back-translated data to train a range of NMT systems for German- c⃝ 2018 The authors. This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CC- BY-ND. to-English, and analyse the resulting trans- lation performance. 1 Introduction Neural Machine Translation (NMT) [Cho et al., 2014; Sutskever et al., 2014; Bahdanau et al., 2015] is a relatively new machine translation (MT) paradigm that has quickly become dominant in both academic and industry MT communities, achieving state-of-the-art results [Bentivogli et al., 2016; Bojar et al., 2016; Junczys-Dowmunt et al., 2016; Wu et al., 2016; Castilho et al., 2017; Shte- rionov et al., 2017] on a range of language pairs and domains. As a corpus-based paradigm, the translation quality strongly depends on the qual- ity and quantity of the training data provided. In comparison to statistical machine translation (SMT) [Koehn, 2010], NMT typically requires more data to build a system with good translation performance [Koehn and Knowles, 2017]. In many use-cases, however, the amount of good-quality parallel data available is insufficient to reach the translation standard required. In such cases, it has become the norm to resort to back-translating freely available monolingual data [Sennrich et al., 2016b; Belinkov and Bisk, 2017; Domhan and Hieber, 2017] to create an additional synthetic parallel corpus [Sennrich et al., 2016b] for training an NMT model. In this paper, we assert that this scenario has become the default in NMT without proper con- sideration of the merits of the approach. For example, Rarrick et al. [2011] present an algo- rithm for filtering noisy content from Web-scraped parallel corpora, in order to mitigate the “pol- …

Content truncated to 500 words