Human or Neural Translation?

Proceedings of the 28th International Conference on Computational Linguistics, pages 6553–6564 Barcelona, Spain (Online), December 8-13, 2020 6553 Human or Neural Translation? Shivendra Bhardwaj David Alfonso-Hermelo Philippe Langlais RALI/DIRO Universit´e de Montr´eal Montreal (Quebec) H3C 3J7, Canada felipe@iro.umontreal.ca Gabriel Bernier-Colborne Michel Simard Cyril Goutte National Research Council Canada 1200 Montreal Road, Building M-58 Ottawa, Ontario K1A 0R6, Canada Cyril.Goutte@nrc-cnrc.gc.ca Abstract Deep neural models tremendously improved machine translation. In this context, we investigate whether distinguishing machine from human translations is still feasible. We trained and applied 18 classifiers under two settings: a monolingual task, in which the classifier only looks at the (French) translation; and a bilingual task, in which the source text (in English) is also taken into consideration. We report on extensive experiments involving 4 neural MT systems (Google Translate, DeepL, as well as two systems we trained) and varying the domain of texts. We show that the bilingual task is the easiest one and that transfer-based deep-learning classifiers perform best, with mean accuracies around 85% in-domain and 75% out-of-domain. 1 Introduction This work addresses the task of distinguishing between translations produced by humans and machines. Practical applications for this include: improving machine translation systems (Li et al., 2015), filtering parallel data mined from the Web (Arase and Zhou, 2013) and evaluating machine translation quality without reference translations (Aharoni et al., 2014). In our case, we are more interested in tracing the origin of translations outsourced by a large institutional translation service. Our work aims at distinguishing between human and neural machine translations at the sentence level. We consider two settings: a monolingual task, where only the target sentence is considered; and a bilin- gual task where both the source text and its translation are available. We compare feature-based ap- proaches with several deep learning methods, investigating the impact of text domains and MT systems (in-house neural engines, Google Translate, DeepL), paying attention to cases where the translation en- gine at test time is different from the one used for training, which we found often not studied in related work. We show that identifying machine translation is still feasible nowadays. On the bilingual task, the best transfer learning method we tested recorded an in-domain accuracy of 87.6% and out-of-domain performances ranging between 65.4% and 84.2% depending on the domain of texts and MT system considered. We analyze why our classifiers manage to do better than chance even though translations produced automatically seem to us of very good quality overall. We believe our study offers many new data points, and hope it will foster research on this timely topic. After reviewing related work in Section 2, we describe our dataset and experimental setting in Sec- tion 3, the neural MT systems we used in our experiments in Section 4 and the classifiers we tested in Section 5. We present our experimental results in Section 6 and propose a deeper analysis in Section 7. 2 Related Work Most studies on identifying machine translation were conducted at a time where MT systems …

Content truncated to 500 words