Translationese as a Language in “Multilingual” NMT

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7737–7746 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 7737 Translationese as a Language in “Multilingual” NMT Parker Riley△∗, Isaac Caswell▽, Markus Freitag▽, David Grangier▽ △University of Rochester ▽Google Research Abstract Machine translation has an undesirable propensity to produce “translationese” ar- tifacts, which can lead to higher BLEU scores while being liked less by human raters. Motivated by this, we model translationese and original (i.e. natural) text as separate languages in a multilingual model, and pose the question: can we perform zero-shot translation between original source text and original target text? There is no data with original source and original target, so we train a sentence-level classifier to distinguish translationese from original target text, and use this classifier to tag the training data for an NMT model. Using this technique we bias the model to produce more natural outputs at test time, yielding gains in human evaluation scores on both adequacy and fluency. Addi- tionally, we demonstrate that it is possible to bias the model to produce translationese and game the BLEU score, increasing it while decreasing human-rated quality. We analyze these outputs using metrics measuring the degree of translationese, and present an analysis of the volatility of heuristic-based train-data tagging. 1 Introduction “Translationese” is a term that refers to artifacts present in text that was translated into a given lan- guage that distinguish it from text originally written in that language (Gellerstam, 1986). These artifacts include lexical and word order choices that are in- fluenced by the source language (Gellerstam, 1996) as well as the use of more explicit and simpler constructions (Baker et al., 1993). These differences between translated and origi- nal text mean that the direction in which parallel data (bitext) was translated is potentially impor- tant for machine translation (MT) systems. Most *Work done while at Google Research. Translated Original Translated Original Target Source Src-Orig Data Trg-Orig Data MT Training Bitext Figure 1: Illustration of MT train+test parallel data, or- ganized into quadrants based on whether the source or target is translated or original. parallel data is either source-original (the source was translated into the target) or target-original (the target was translated into the source), though some- times neither side is original because both were translated from a third language. Figure 1 illustrates the four possible combina- tions of translated and original source and target data. Recent work has examined the impact of translationese in MT evaluation, using the WMT evaluation campaign as the most prominent exam- ple. From 2014 through 2018, WMT test sets were constructed such that 50% of the sentence pairs are source-original (upper right quadrant of Figure 1) and the rest are target-original (lower left quadrant). Toral et al. (2018), Zhang and Toral (2019), and Graham et al. (2019) have examined the effect of this testing setup on MT evaluation, and have all argued that target-original test data should not be included in future evaluation campaigns because …

Content truncated to 500 words