Like Chalk and Cheese? On the Effects of Translationese in MT Training

Like Chalk and Cheese? On the Effects of Translationese in MT Training Samuel Larkin Samuel.Larkin@nrc-cnrc.gc.ca Michel Simard Michel.Simard@nrc-cnrc.gc.ca Rebecca Knowles Rebecca.Knowles@nrc-cnrc.gc.ca National Research Council Canada, Ottawa, Ontario, Canada Abstract We revisit the topic of translation direction in the data used for training neural machine trans- lation systems, focusing on a real-world scenario with known translation direction and im- balances in translation direction: the Canadian Hansard. According to automatic metrics, we observe that using parallel data that was produced in the “matching” translation direction (au- thentic source, translationese target) improves translation quality. In cases of data imbalance in terms of translation direction, we find that tagging the translation direction of training data can close the performance gap. We perform a human evaluation that differs slightly from the automatic metrics, but nevertheless confirms that for this French–English dataset that is known to contain high-quality translations, authentic or tagged mixed source improves over transla- tionese source for training. 1 Introduction Prior work in statistical machine translation (SMT) highlighted potential benefits of making use of information about the translation direction of training data (Kurokawa et al., 2009). When text is translated, there is an authentic source (the language in which the text was originally produced), and its translation, which in contrast can be described as translationese. Thus when considering translation direction in machine translation, training data can be described as con- sisting of authentic source, translationese source, or a mix.1 Backtranslated data produced by machine translation may be thought of as an extreme case of translationese source (Marie et al., 2020), but because the quality and types of errors that occur in machine translation are quite different from those that occur in human translation, it is worth examining translation direction of human translation separately from MT-based data augmentation. In Figure 1 we show a fairly dramatic example of the kinds of translation quality differences that can occur when building MT systems using authentic source as opposed to translationese source. Recent work in neural machine translation (NMT) has revisited this issue, motivating the automatic detection of (human) translationese by showing improved performance on several metrics when training translation direction matches the testing translation direction (Sominsky and Wintner, 2019), examining domain and backtranslation along with the translation direction of test sets (Bogoychev and Sennrich, 2019), and evaluating the treatment of predicted trans- lation direction as separate languages in a multilingual-style NMT system through human and automatic metrics (Riley et al., 2020). 1For the purposes of this paper, we will set aside the situation where both sides of the text consist of translationese, translated from one or more other pivot languages. Proceedings of the 18th Biennial Machine Translation Summit Virtual USA, August 16 - 20, 2021, Volume 1: MT Research Track Page 103 Source Les producteurs de fromage au Qu´ebec sont des fleurons dont on est fiers. Reference We are proud of our exceptional Quebec cheese producers. MT (Authentic Src.) We are proud of the success of cheese producers in Quebec. MT (Translationese Src.) The cheese producers …

Content truncated to 500 words