Machine Translated Text Detection Through Text Similarity with Round-Trip Translation

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5792–5797 June 6–11, 2021. ©2021 Association for Computational Linguistics 5792 Machine Translated Text Detection Through Text Similarity with Round-Trip Translation Hoang-Quoc Nguyen-Son 1, Tran Phuong Thao 2, Seira Hidano 1, Ishita Gupta 3, and Shinsaku Kiyomoto 1 1KDDI Research, Inc., Japan 2The University of Tokyo, Japan 3Indian Institute of Technology, India 1{ho-nguyen,se-hidano,kiyomoto}@kddi-research.jp 2tpthao@yamagula.ic.i.u-tokyo.ac.jp 3ishita.gupta.ee317@ee.iitd.ac.in Abstract Translated texts have been used for malicious purposes, i.e., plagiarism or fake reviews. Ex- isting detectors have been built around a spe- cific translator (e.g., Google) but fail to de- tect a translated text from a strange transla- tor. If we use the same translator, the trans- lated text is similar to its round-trip translation, which is when text is translated into another language and translated back into the origi- nal language. However, a round-trip translated text is significantly different from the original text or a translated text using a strange trans- lator. Hence, we propose a detector using text similarity with round-trip translation (TSRT). TSRT achieves 86.9% accuracy in detecting a translated text from a strange translator. It outperforms existing detectors (77.9%) and hu- man recognition (53.3%). 1 Introduction A reader may misunderstand the original meaning of a translated text1. For example, Facebook trans- lated “good morning” into “attack them,” leading to an arrest2. Adversaries can use a translator for ma- licious tasks such as round-trip translation used in plagiarism (Jones and Sheridan, 2015) to avoid hu- man recognition or in adversarial text (Iyyer et al., 2018) to fool AI. Existing work has investigated the detection of translated texts in various approaches. The parse tree approach (Chae and Nenkova, 2009; Li et al., 2015) exploits text structure. The N- gram approach (Aharoni et al., 2014; Arase and Zhou, 2013) estimates text fluency. The text com- plexity approach uses complex words (Nguyen- 1When we mention a translated text, translation, translator, and Google, all are related to machine translation systems 2www.theguardian.com/technology/2017/oct/24/facebook- palestine-israel-translates-good-morning-attack-them-arrest Son and Echizen, 2017) and phrases (Nguyen- Son et al., 2017). The text coherence approach is based on matching similar words on a paragraph level (Nguyen-Son et al., 2018, 2019b). A three- layer CNN (Riley et al., 2020) is trained on ei- ther one-way or round-trip translated texts. Our previous work (Nguyen-Son et al., 2019a) com- bined round-trip translation with BLEU scores. All these approaches fail to detect a text translated by another translator or from a different language. Motivation The first translation round induces a low similarity between the translated and original texts, whereas the extent of similarity increases in later rounds (Vanmassenhove et al., 2019). Let us consider an example in Fig. 1. We randomly se- lected an English text t from an English-Russian pair3; the Russian text was translated into En- glish by Google, called t′(Go,RU→EN). We mea- sured the similarity between a text and its round- trip translation using the minimum edit distance (MED) (Levenshtein, 1966). The translated text t′ …

Content truncated to 500 words