Agree to Disagree: Analysis of Inter-Annotator Disagreements in Human Evaluation of Machine Translation Output

Proceedings of the 25th Conference on Computational Natural Language Learning (CoNLL), pages 234–243 November 10–11, 2021. ©2021 Association for Computational Linguistics 234 Agree to Disagree: Analysis of Inter-Annotator Disagreements in Human Evaluation of Machine Translation Output Maja Popovi´c ADAPT Centre School of computing Dublin City University, Ireland maja.popovic@adaptcentre.ie Abstract This work describes an analysis of inter- annotator disagreements in human evaluation of machine translation output. The errors in the analysed texts were marked by multiple annota- tors under guidance of different quality criteria: adequacy, comprehension, and an unspecified generic mixture of adequacy and fluency. Our results show that different criteria result in dif- ferent disagreements, and indicate that a clear definition of quality criterion can improve the inter-annotator agreement. Furthermore, our results show that for certain linguistic phenomena which are not limited to one or two words (such as word ambigu- ity or gender) but span over several words or even entire phrases (such as negation or relative clause), disagreements do not necessarily repre- sent “errors" or “noise" but are rather inherent to the evaluation process. On the other hand, for some other phenomena (such as omission or verb forms) agreement can be easily improved by providing more precise and detailed instruc- tions to the evaluators. 1 Introduction Despite the large number of automatic evaluation metrics designed for machine translation (MT) eval- uation1 which represent invaluable tools for rapid development of MT systems, human assessment of translation quality remains the gold standard, both for evaluating MT systems as well as for develop- ing suitable automatic metrics. Human evaluation is typically provided in one of the following ways: assigning an overall quality score to each translated sentence, ranking two or more translations of the same source language sentence from best to worst, or annotating actual translation errors. The errors can be only highlighted (marked) or corrected (post- edited), but can also be classified according to a 1http://www.statmt.org/wmt20/ metrics-task.html given pre-defined scheme, such as MQM scheme.2 However, human judgments of translation qual- ity show a high degree of variance (Callison-Burch et al., 2008; Denkowski and Lavie, 2010), espe- cially for fine-grained error classification based on a detailed error scheme involving many error types (Lommel et al., 2014; Klubiˇcka et al., 2018). One of the reasons for the variance is the variety of possible solutions: there is no single objectively correct translation of a given text, but rather a range of possible translations from perfect over good to acceptable. Moreover, there is no single univer- sal criterion for translation quality. Although all manual evaluations are essentially based on some of the following three criteria: adequacy (mean- ing preservation; the most frequently used), flu- ency (grammar of the target language; frequently used) and comprehension (readability; rarely used), the precise definition of the criterion is not always given to the annotators. Very often, an unspecified mixture of adequacy and fluency is used. In ad- dition, other factors like target audience, goal of translation, etc. can have influence on evaluator’s perception of quality. Most publications dealing …

Content truncated to 500 words