Tagged Back-Translation

Proceedings of the Fourth Conference on Machine Translation (WMT), Volume 1: Research Papers, pages 53–63 Florence, Italy, August 1-2, 2019. c⃝2019 Association for Computational Linguistics 53 Tagged Back-Translation Isaac Caswell, Ciprian Chelba, David Grangier Google Research {icaswell, ciprianchelba, grangier}@google.com Abstract Recent work in Neural Machine Translation (NMT) has shown significant quality gains from noised-beam decoding during back- translation, a method to generate synthetic par- allel data. We show that the main role of such synthetic noise is not to diversify the source side, as previously suggested, but simply to indicate to the model that the given source is synthetic. We propose a simpler alternative to noising techniques, consisting of tagging back-translated source sentences with an ex- tra token. Our results on WMT outperform noised back-translation in English-Romanian and match performance on English-German, re-defining state-of-the-art in the former. 1 Introduction Neural Machine Translation (NMT) has made considerable progress in recent years (Bahdanau et al., 2015; Gehring et al., 2017; Vaswani et al., 2017). Traditional NMT has relied solely on par- allel sentence pairs for training data, which can be an expensive and scarce resource. This moti- vates the use of monolingual data, usually more abundant (Lambert et al., 2011). Approaches using monolingual data for machine translation include language model fusion for both phrase- based (Brants et al., 2007; Koehn, 2009) and neural MT (G¨ulc¸ehre et al., 2015, 2017), back- translation (Sennrich et al., 2016; Poncelas et al., 2018), unsupervised machine translation (Lample et al., 2018a; Artetxe et al., 2018), dual learning (Cheng et al., 2016; Di He and Ma, 2016; Xia et al., 2017), and multi-task learning (Domhan and Hieber, 2017). We focus on back-translation (BT), which, de- spite its simplicity, has thus far been the most effective technique (Sennrich et al., 2017; Ha et al., 2017; Garc´ıa-Mart´ınez et al., 2017). Back- translation entails training an intermediate target- to-source model on genuine bitext, and using this model to translate a large monolingual corpus from the target into the source language. This al- lows training a source-to-target model on a mix- ture of genuine parallel data and synthetic pairs from back-translation. We build upon Edunov et al. (2018) and Ima- mura et al. (2018), who investigate BT at the scale of hundreds of millions of sentences. Their work studies different decoding/generation meth- ods for back-translation: in addition to regular beam search, they consider sampling and adding noise to the one-best hypothesis produced by beam search. They show that sampled BT and noised- beam BT significantly outperform standard BT, and attribute this success to increased source-side diversity (sections 5.2 and 4.4). Our work investigates noised-beam BT (NoisedBT) and questions the role noise is play- ing. Rather than increasing source diversity, our work instead suggests that the performance gains come simply from signaling to the model that the source side is back-translated, allowing it to treat the synthetic parallel data differently than the natural bitext. We hypothesize that BT introduces both helpful signal (strong target-language signal and weak cross-lingual signal) and harmful signal (amplifying …

Content truncated to 500 words