Original or Translated? A Causal Analysis of the Impact of Translationese on Machine Translation Performance

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5303 - 5320 July 10-15, 2022 ©2022 Association for Computational Linguistics Original or Translated? A Causal Analysis of the Impact of Translationese on Machine Translation Performance Jingwei Ni∗ University College London ucabjni@ucl.ac.uk Zhijing Jin∗ MPI & ETH Zürich zjin@tue.mpg.de Markus Freitag Google Research freitag@google.com Mrinmaya Sachan ETH Zürich msachan@ethz.ch Bernhard Schölkopf MPI & ETH Zürich bs@tue.mpg.de Abstract Human-translated text displays distinct fea- tures from naturally written text in the same language. This phenomena, known as transla- tionese, has been argued to confound the ma- chine translation (MT) evaluation. Yet, we find that existing work on translationese ne- glects some important factors and the conclu- sions are mostly correlational but not causal. In this work, we collect CAUSALMT, a dataset where the MT training data are also labeled with the human translation directions. We in- spect two additional critical factors, the train- test direction match (whether the human trans- lation directions in the training and test sets are aligned), and data-model direction match (whether the model learns in the same direc- tion as the human translation direction in the dataset). We show that these two factors have a large causal effect on the MT performance, in addition to the test-model direction mismatch highlighted by existing work on translationese. In light of our findings, we provide a set of suggestions for MT training and evaluation.1 1 Introduction MT has long been concerned with the artifacts in- troduced by translationese, the human-translated text that is systematically different from naturally written text in the same language (Toury, 1980; Gellerstam, 1986; Toury, 1995; Baker, 1993; Ba- roni and Bernardini, 2006). For a translation sys- tem translating from language X to language Y , there can be two types of test data: sentences that originated in language X and are human-translated into language Y (denoted as X H −→Y ), and sen- tences that originated in language Y and human- translated into language X (denoted as X H ←−Y ).2 ∗Equal contributions. 1Our code and data are at https://github.com/ EdisonNi-hku/CausalMT. 2Note that the scope of this work does not cover pivot trans- lation through a third language, but we encourage exploration in future work. Real Translation Quality MT Performance Factor 1: Translationese in the Test Set (Test-Model Direction Match) Factor 2: Train-Test Direction Match Factor 3: Data-Model Direction Match Most Studies Our work Previous work: Correlational studies on some of the factors (mostly Factor 1). Our work: A systematic causal analysis of the additional causal effects of Factor 2 and Factor 3. Figure 1: Three different factors illustrate the impact of translationese on MT performance. Previous work mainly focuses on how translationese in the test set (Factor 1) inflates BLEU score and makes it favor some translation systems over others. Our work investigates the causal effects of the other two key factors, the train- test direction match (Factor 2; i.e., whether the training set and the …

Content truncated to 500 words