Document-Level Machine Translation Evaluation Project: Methodology, Effort and Inter-Annotator Agreement

Document-Level Machine Translation Evaluation Project: Methodology, Effort and Inter-Annotator Agreement Sheila Castilho ADAPT Centre School of Computing Dublin City University sheila.castilho@adaptcentre.ie Abstract Recently, document-level (doc-level) hu- man evaluation of machine translation (MT) has raised interest in the commu- nity after a few attempts have disproved claims of “human parity” (Toral et al., 2018; L¨aubli et al., 2018). However, lit- tle is still known about best practices re- garding doc-level human evaluation. This project aims to identify methodologies to better cope with i) the current state-of-the- art (SOTA) human metrics, ii) a possible complexity when assigning a single score to a text consisted of ‘good’ and ‘bad’ sentences, iii) a possible tiredness bias in doc-level set-ups, and iv) the difference in inter-annotator agreement (IAA) between sentence and doc-level set-ups. 1 Introduction Although currently an active community is work- ing on developing document-level (doc-level) MT systems, their evaluation has primarily been per- formed at the sentence level. In 2019, for the first time, WMT19 attempted a doc-level human evalu- ation for the news domain, after considering criti- cisms by Toral et al. (2018) and L¨aubli et al. (2018) regarding the current best practices in MT evalu- ation. Both papers independently reassessed the claims of MT “achieving human parity” and found that the lack of extra-sentential context has a great effect on quality assessment. In a recent survey with native speakers, Castilho et al. (2020) tested the context span for the transla- tion of three different domains (reviews, subtitles, c⃝ 2020 The authors. This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CC- BY-ND. and literature). Results show that over 33% of the sentences tested (300 in total) required more con- tent than the sentence itself to be translated, and from those, 23% required more than two previous sentences to be properly translated. Some of the issues which the participants found to most hinder the translation include word ambiguity, terminol- ogy, and gender agreement. Moreover, the authors found that there are differences in issues and con- text span between domains. This shows that doc- level evaluation enables to assess suprasentential context, textual cohesion and coherence types of errors. In one of the few studies on doc-level evalua- tion, L¨aubli et al. (2018) use pairwise rankings of fluency and adequacy in which raters give one sin- gle score to the full document. For WMT19, the direct assessment task asked crowdworkers to give a single score (0–100) to full documents for ac- curacy, where only one MT output is shown each time (no comparison with other MT system). With that in mind, this project aims at identify- ing methodologies to better cope with the SOTA human metrics, namely ratings of fluency and ad- equacy, error mark up and ranking evaluations (Castilho et al., 2018). We will gauge the complex- ity when assigning a single score to full texts, since they can consist of ‘good’ and ‘bad’ sentences, which could mean that instead of a single score, translators would prefer to …

Content truncated to 500 words