The Effect of Translationese in Machine Translation Test Sets

Proceedings of the Fourth Conference on Machine Translation (WMT), Volume 1: Research Papers, pages 73–81 Florence, Italy, August 1-2, 2019. c⃝2019 Association for Computational Linguistics 73 The Effect of Translationese in Machine Translation Test Sets Mike Zhang Information Science Programme University of Groningen The Netherlands j.j.zhang.1@student.rug.nl Antonio Toral Center for Language and Cognition University of Groningen The Netherlands a.toral.ruiz@rug.nl Abstract The effect of translationese has been studied in the field of machine translation (MT), mostly with respect to training data. We study in depth the effect of translationese on test data, using the test sets from the last three editions of WMT’s news shared task, containing 17 translation directions. We show evidence that (i) the use of translationese in test sets results in inflated human evaluation scores for MT systems; (ii) in some cases system rankings do change and (iii) the impact translationese has on a translation direction is inversely cor- related to the translation quality attainable by state-of-the-art MT systems for that direction. 1 Introduction Translated texts in a human language exhibit unique characteristics that set them apart from texts originally written in that language. It is common then to refer to translated texts with the term translationese. The characteristics of trans- lationese can be grouped along the so-called uni- versal features of translation or translation univer- sals (Baker, 1993), namely simplification, normal- isation and explicitation. In addition to these three, interference is recognised as a fundamental law of translation (Toury, 2012): “phenomena pertaining to the make-up of the source text tend to be trans- ferred to the target text”. In a nutshell, compared to original texts, translations tend to be simpler, more standardised, and more explicit and they re- tain some characteristics that pertain to the source language. The effect of translationese has been studied in machine translation (MT), mainly with respect to the training data, during the last decade. Previous work has found that an MT system performs better when trained on parallel data whose source side is original and whose target side is translationese, rather than the opposite (Kurokawa et al., 2009; Lembersky, 2013). A recent paper has studied the effect of transla- tionese on test sets (Toral et al., 2018), in the con- text of assessing the claim of human parity made on Chinese-to-English WMT’s 2017 test set (Has- san et al., 2018). The source side of this test set, as it is common in WMT (Bojar et al., 2016, 2017, 2018), was half original and half translationese. It was found out that the translationese part was ar- tificially easier to translate, which resulted in in- flated scores for MT systems. Noting that this finding was based on one test set for a single translation direction, we explore this topic in more depth, studying the effect of translationese in all the language pairs of the news shared task of WMT 2016 to 2018. Our research questions (RQs) are the following: • RQ1. Does the use of translationese in the source side of MT test sets …

Content truncated to 500 words