On The Evaluation of Machine Translation Systems Trained With Back-Translation

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2836–2846 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 2836 On The Evaluation of Machine Translation Systems Trained With Back-Translation Sergey Edunov Myle Ott Marc’Aurelio Ranzato Michael Auli Facebook AI Research Abstract Back-translation is a widely used data augmen- tation technique which leverages target mono- lingual data. However, its effectiveness has been challenged since automatic metrics such as BLEU only show significant improvements for test examples where the source itself is a translation, or translationese. This is be- lieved to be due to translationese inputs bet- ter matching the back-translated training data. In this work, we show that this conjecture is not empirically supported and that back- translation improves translation quality of both naturally occurring text as well as transla- tionese according to professional human trans- lators. We provide empirical evidence to sup- port the view that back-translation is preferred by humans because it produces more fluent outputs. BLEU cannot capture human pref- erences because references are translationese when source sentences are natural text. We recommend complementing BLEU with a lan- guage model score to measure fluency. 1 Introduction Back-translation (BT; Bojar and Tamchyna 2011; Sennrich et al. 2016a; Poncelas et al. 2018a) is a data augmentation method that is a key ingre- dient for improving translation quality of neural machine translation systems (NMT; Sutskever et al. 2014; Bahdanau et al. 2015; Gehring et al. 2017; Vaswani et al. 2017). NMT systems using large- scale BT have been ranked top at recent WMT evaluation campaigns (Bojar et al., 2018; Edunov et al., 2018; Ng et al., 2019). The idea is to train a target-to-source model to generate additional syn- thetic parallel data from monolingual target data. The resulting sentence pairs have synthetic sources and natural targets which are then added to the original bitext in order to train the desired source- to-target model. BT improves generalization and can be used to adapt models to the test domain by adding appropriate monolingual data. Parallel corpora are usually comprised of two types of sentence-pairs: sentences which originate in the source language and have been translated by humans into the target language, or sentences which originate from the target language and have been translated into the source language. We refer to the former as the direct portion and the latter as the reverse portion. The setup we are ultimately in- terested in is models that translate direct sentences. Translations produced by human translators, or translationese tend to be simpler and more standardized compared to naturally occurring text (Baker, 1993; Zhang and Toral, 2019; Toury, 2012). Several recent studies found that such re- verse test sentences are easier to translate than di- rect sentences (Toral et al., 2018; Graham et al., 2019), and human judges consistently assign higher ratings to translations of target original sentences than to source original sentences. These studies therefore recommend to restrict test sets to source original sentences, a methodology which has been adopted by …

Content truncated to 500 words