Identifying Computer-Translated Paragraphs using Coherence Features

Identifying Computer-Translated Paragraphs using Coherence Features Hoang-Quoc Nguyen-Son 1,2, Ngoc-Dung T. Tieu 3, Huy H. Nguyen 3, Junichi Yamagishi 1,3,4, and Isao Echizen 1,3 1National Institute of Informatics, Tokyo, Japan 2KDDI Research Inc., Saitama, Japan 3The Graduate University for Advanced Studies, Kanagawa, Japan 4The University of Edinburgh, Edinburgh, United Kingdom {nshquoc,nhhuy,dungtieu,jyamagis,iechizen}@nii.ac.jp Abstract We have developed a method for extracting the coherence features from a paragraph by matching similar words in its sentences. We conducted an experiment with a parallel Ger- man corpus containing 2000 human-created and 2000 machine-translated paragraphs. The result showed that our method achieved the best performance (accuracy = 72.3%, equal error rate = 29.8%) when it is compared with previous methods on various computer- generated text including translation and paper generation (best accuracy = 67.9%, equal error rate = 32.0%). Experiments on Dutch, another rich resource language, and a low resource one (Japanese) attained similar performances. It demonstrated the efficiency of the coherence features at distinguishing computer-translated from human-created paragraphs on diverse languages. 1 Introduction Computer-translated text plays an essential role in modern life. Such artificial text helps people, who use different languages, can communicate each other. Machine-translated systems thus significantly support or even completely relieve human transla- tors and interpreters from time-consuming burden. Thanks to deep learning, neural machine transla- tion have been drastically progressed recently. How- ever, we can still tell “this text must be automati- cally translated by a computer and reads strangely” sometimes. The unexpected quality leads readers to confuse or misunderstand the meaning of artifi- cial text comparing with the original meaning such as machine-translated web pages, especially in low resource languages. Enhanced methods are thus needed to identify machine-translated text. Research on computer-translated text detection has been of interest to the natural language process- ing community. Most detection methods are aimed at the sentence level and use a tree parser (Chae and Nenkova, 2009; Li et al., 2015) to estimate the naturalness of a text passage. However, the scope of these methods is limited to individual sentences and ignores the relationships among sentences. In contrast, some methods are aimed at identifying the translation by the POS N-gram model (Arase and Zhou, 2013; Aharoni et al., 2014), but they extract features from only a few adjacent words. Other methods identify the translated documents (Nguyen- Son et al., 2017) or generated papers (Labb´e and Labb´e, 2013) using word distribution. However, such methods are only suitable for huge text. Of the various levels of text, i.e., word, phrase, sentence, paragraph and document, the paragraph is one of the most important. For instance, paragraphs help readers to quickly receive important content amidst large pieces of text on various topics, such as abstracts of scholarly papers and news summaries. Moreover, the paragraph is the main part of current digital text (e.g., email and online product descrip- tions). The paragraph level provides more informa- tion compared with the sentence level. A large doc- ument is often separated into individual paragraphs whose sentences have meanings in common and …

Content truncated to 500 words