On the Same Page? Comparing Inter-Annotator Agreement in Sentence and Document Level Human Machine Translation Evaluation

Proceedings of the 5th Conference on Machine Translation (WMT), pages 1150–1159 Online, November 19–20, 2020. c⃝2020 Association for Computational Linguistics 1150 On the Same Page? Comparing Inter-Annotator Agreement in Sentence and Document Level Human Machine Translation Evaluation Sheila Castilho School of Computing Adapt Centre Dublin City University sheila.castilho@adaptcentre.ie Abstract Document-level evaluation of machine transla- tion has raised interest in the community es- pecially since responses to the claims of “hu- man parity” (Toral et al., 2018; L¨aubli et al., 2018) with document-level human evaluations have been published. Yet, little is known about best practices regarding human evalua- tion of machine translation at the document- level. This paper presents a comparison of the differences in inter-annotator agreement between quality assessments using sentence and document-level set-ups. We report results of the agreement between professional transla- tors for fluency and adequacy scales, error an- notation, and pair-wise ranking, along with the effort needed to perform the different tasks. To best of our knowledge, this is the first study of its kind. 1 Introduction Increasing efforts have been made in order to add discourse into neural machine translation (NMT) systems. However, the results reported for those attempts are somehow limited as the evaluation is still mostly performed at the sentence level, us- ing single references, which are not able to recog- nise the improvements of those systems. The state- of-the-art automatic evaluation metrics have been shown to underestimate the quality of NMT sys- tems (Shterionov et al., 2018), and the suitability of these metrics for document-level systems has also been criticised (Smith, 2017). For that reason, document-level human evaluation of machine trans- lation (MT) has raised interest in the community recently as it enables the assessment of suprasen- tential context. In a survey with native speakers, Castilho et al. (2020) tested the context span for the translation of 300 sentences in three different domains, namely reviews, subtitles, and literature. Over 33% of the sentences tested were found to require more con- text than the sentence itself to be translated, and from those, 23% required more than two previ- ous sentences to be properly translated. Ambigu- ity, terminology, and gender agreement were the most common issues found to hinder translation. Moreover, differences in issues and context span were found between domains. This shows that document-level evaluation enables the assessment of textual cohesion and coherence types of errors which are impossible at times to recognise at sen- tence level. Recent attempts to assess quality at the document-level were described in Toral et al. (2018) and L¨aubli et al. (2018) who independently reassessed the bold claims of MT ‘achieving human parity’ and found that the lack of extra-sentential context has a great effect on quality assessment, and pointed to a failure of the current best prac- tices in MT evaluation. Toral et al. (2018) used consecutive single sentences to rank translations by two MT systems and a human reference. They found that the evaluators were able to better assess the translations when provided with more context, and …

Content truncated to 500 words