Relations between comprehensibility and adequacy errors in machine translation output

Proceedings of the 24th Conference on Computational Natural Language Learning, pages 256–264 Online, November 19-20, 2020. c⃝2020 Association for Computational Linguistics https://doi.org/10.18653/v1/P17 256 Relations between comprehensibility and adequacy errors in machine translation output Maja Popovi´c ADAPT Centre, School of Computing Dublin City University, Ireland maja.popovic@adaptcentre.ie Abstract This work presents a detailed analysis of trans- lation errors perceived by readers as compre- hensibility and/or adequacy issues. The main finding is that good comprehensibility, sim- ilarly to good fluency, can mask a number of adequacy errors. Of all major adequacy errors, 30% were fully comprehensible, thus fully misleading the reader to accept the in- correct information. Another 25% of ma- jor adequacy errors were perceived as almost comprehensible, thus being potentially mis- leading. Also, a vast majority of omissions (about 70%) is hidden by comprehensibility. Further analysis of misleading translations re- vealed that the most frequent error types are ambiguity, mistranslation, noun phrase error, word-by-word translation, untranslated word, subject-verb agreement, and spelling error in the source text. However, none of these error types appears exclusively in misleading trans- lations, but are also frequent in fully incorrect (incomprehensible inadequate) and discarded correct (incomprehensible adequate) transla- tions. Deeper analysis is needed to potentially detect underlying phenomena specifically re- lated to misleading translations. 1 Introduction While automatic evaluation metrics are very impor- tant and invaluable tools for rapid development of machine translation (MT) systems, they are only a substitution for human assessment of translation quality. Various methods have been proposed and used for the human evaluation of MT quality by as- signing overall scores to MT outputs, such as (AL- PAC, 1966; White et al., 1994; Koehn and Monz, 2006; Callison-Burch et al., 2007; Roturier and Bensadoun, 2011; Graham et al., 2013; Barrault et al., 2019), and all of them rely on at least one of the three translation quality criteria: comprehen- sibility (comprehension, intelligibility), adequacy (fidelity, semantic accuracy), and fluency (gram- maticality). Comprehensibility reflects the degree to which a translated text can be understood, ade- quacy reflects the degree to which the translation conveys the meaning of the original text in the source language, and fluency reflect the grammar of the translated text. The raters are usually asked to assign an overall score for the given translation criterion. In order to get more details about transla- tion performance, error classification and analysis emerged in the field of MT (Vilar et al., 2006; Lom- mel et al., 2014; Klubiˇcka et al., 2018; Van Brussel et al., 2018). However, there is less work dealing with human perception of MT quality and errors. For statistical phrase-based MT systems (SMT), Kirchhoff et al. (2014) and Federico et al. (2014) were identifying error types which are mostly disliked by readers. In the last five years, systems based on artificial neural networks (NMT) have become the new state of the art. Several evaluation studies, such as (Castilho et al., 2017; Klubiˇcka et al., 2018; Van Brussel et al., 2018) reported that these systems are able to produce more fluent and …

Content truncated to 500 words