Automatic Discrimination of Human and Neural Machine Translation in Multilingual Scenarios

Automatic Discrimination of Human and Neural Machine Translation in Multilingual Scenarios Malina Chichirau Bernoulli Institute University of Groningen m.chichirau@student.rug.nl Rik van Noord CLCG University of Groningen rikvannoord@gmail.com Antonio Toral CLCG University of Groningen a.toral.ruiz@rug.nl Abstract We tackle the task of automatically dis- criminating between human and machine translations. As opposed to most previ- ous work, we perform experiments in a multilingual setting, considering multiple languages and multilingual pretrained lan- guage models. We show that a classifier trained on parallel data with a single source language (in our case German–English) can still perform well on English transla- tions that come from different source lan- guages, even when the machine transla- tions were produced by other systems than the one it was trained on. Additionally, we demonstrate that incorporating the source text in the input of a multilingual classifier improves (i) its accuracy and (ii) its robust- ness on cross-system evaluation, compared to a monolingual classifier. Furthermore, we find that using training data from mul- tiple source languages (German, Russian, and Chinese) tends to improve the accu- racy of both monolingual and multilingual classifiers. Finally, we show that bilin- gual classifiers and classifiers trained on multiple source languages benefit from be- ing trained on longer text sequences, rather than on sentences. 1 Introduction In many NLP tasks one may want to filter out ma- chine translations (MT), but keep human transla- tions (HT). Consider, for example, the construc- © 2023 The authors. This article is licensed under a Creative Commons 4.0 licence, no derivative works, attribution, CC- BY-ND. tion of parallel corpora used for training MT sys- tems: filtering out MT output is getting progres- sively harder, given the ever-increasing quality of neural MT (NMT) systems. Moreover, the exis- tence of such high-quality NMT systems might ag- gravate the problem, as people are getting more likely to employ them when creating texts. In addi- tion, it is also hard to get fair training data to build a classifier that can distinguish between these two types of translations, since publicly-available par- allel corpora with human translations were likely used in the training of well-known publicly avail- able MT systems (such as Google Translate or DeepL). Therefore, we believe that making the most of the scarcely available (multilingual) train- ing data is a crucial research direction. Previous work aiming at discriminating between HT and NMT operates mostly in a monolingual setting (Fu and Nederhof, 2021; van der Werff et al., 2022), To our knowledge, the only ex- ception is Bhardwaj et al. (2020), who targeted English–French, and fine-tuned not only French LMs (monolingual target-only setting), but also multilingual LMs, so that the classifier had also ac- cess to the source text. However, this work used an in-house data set, therefore limiting reproducibil- ity and practical usefulness. There is also older work that tackled statistical MT (SMT) vs HT clas- sification (Arase and Zhou, 2013; Aharoni et al., 2014; Li et al., 2015). Nevertheless, since both the MT paradigm (SMT) and the classifiers used are …

Content truncated to 500 words