Assessing the Comprehensibility of Automatic Translations (ArisToCAT)

Assessing the Comprehensibility of Automatic Translations (ArisToCAT) Lieve Macken, Margot Fonteyne, Arda Tezcan and Joke Daems LT3, Language and Translation Technology Team Ghent University Belgium Lieve.Macken@ugent.be Abstract The ArisToCAT project aims to assess the comprehensibility of ‘raw’ (unedited) MT output for readers who can only rely on the MT output. In this project description, we summarize the main results of the project and present future work. 1 Introduction Machine translation (MT) systems cannot guaran- tee that the text they produce will be fluent and coherent in both syntax and semantics. Errors occur frequently in machine-translated text, leav- ing the reader to guess parts of the intended mes- sage. With the arrival of neural machine transla- tion (NMT), however, the quality of machine trans- lation has increased significantly. As such, ma- chine translation is becoming an attractive solu- tion to deal with the increased need for translated content. This could mean that, in the near future, readers will be more often confronted with ‘raw’ (unedited) MT output. 2 Quality of MT output To assess the quality improvements in MT, we compared the quality of three different MT sys- tems for English–Dutch: a commercial neural sys- tem, a phrase-based system and a predominantly rule-based system. We used Web-Anno1 as anno- tation tool and adopted a two-step approach to an- notate all errors in the MT output. In a first step, only the target text was visible and we marked all fluency errors; in a second step all accuracy errors c⃝ 2020 The authors. This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CC- BY-ND. 1https://webanno.github.io/webanno/ were labelled in both source and target text and were linked. Van Brussel et al. (2018) found that the neural system, in general, outperformed the phrase-based and rule-based systems when con- sidering fluency. The output of the neural system contained fewer grammatical errors and hardly any spelling mistakes. For accuracy, the improvements of NMT are less apparent. The target sentence does not always contain traces of the errors or clues of omissions, which might have an impact on the comprehension. 3 Reading comprehension tests In a pilot study, Macken and Ghyselen (2018) se- lected three texts of the English MT Evaluation version of the Corpus of Reading Comprehension Exercises (Scarton and Specia, 2016) and set up a reading comprehension test for both human trans- lated and raw MT texts. Ninety-nine participants were asked to read the translation very carefully af- ter which they had to answer comprehension ques- tions without having access to the translated text. Human translations received the best overall clar- ity scores, but the reading comprehension tests provided much less unequivocal results. 4 Comprehensibility of newly invented words in NMT output NMT systems occasionally generate non-existing words, i.e words that are not part of the vocabu- lary of the target language and were thus invented by the NMT system. In cases in which readers only have access to the MT output without the source text, such non-existing words can …

Content truncated to 500 words