DELA Project: Document-level Machine Translation Evaluation

DELA Project: Document-level Machine Translation Evaluation Sheila Castilho ADAPT Centre School of Computing Dublin City University sheila.castilho@adaptcentre.ie Abstract This paper presents the results of the Document-level Machine Translation Evaluation (DELA) Project, a two-year project which started in September 2020 funded by the Irish Research Council. This paper describes the results of the project to date, as well as its latest developments. 1 Introduction The challenge of evaluating translations in context has been raising interest in the machine translation (MT) field. However, the definition of what con- stitutes a document-level (doc-level) MT evalua- tion, in terms of how much of the text needs to be shown, is still unclear (Castilho et al., 2020). Few works have taken into account doc-level hu- man evaluation (Barrault et al., 2020), and one common practice is the usage of test suites with context-aware markers. However, test suites with document-level boundaries are still scarce (Rysov´a et al., 2019). The main objective of the DELA Project is to define best practices for doc-level MT evaluation, and test the existing human and auto- matic sentence-level evaluation metrics to the doc- level. We present here the results from the project to date, as well as the upcoming research to be car- ried out. 2 Context Span for MT In Castilho et al. (2020), we tested the context span, that is, the length of context necessary, for the translation of 300 sentences in three differ- ent domains (reviews, subtitles, and literature) and © 2022 The authors. This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CC- BY-ND. showed that over 33% of the sentences tested re- quired more context than the sentence itself to be translated or evaluated, and from those, 23% re- quired more than two previous sentences to be properly evaluated. Ambiguity, terminology, and gender agreement were the most common issues to hinder translation, and moreover, there were ob- servable differences in issues and context span be- tween domains. 3 Doc-Level Evaluation methodology In Castilho (2020; 2021), we tested the differ- ences in inter-annotator agreement (IAA) between single-sentence and doc-level setups. First, trans- lators evaluated the MT output in terms of flu- ency, adequacy, ranking and error annotation in: (i) one score per single isolated sentence, and (ii) one score per document. Then, the doc-level setup was modified, and translators evaluated (i) random single sentences, (ii) individual sentences with ac- cess to the full source and MT output, and (iii) full documents. Results showed that assessing in- dividual sentences within the context of a docu- ment yields a higher IAA compared to the random single-sentence methodology, while when transla- tors give one score per document, IAA is much lower. Assigning one score per sentence in con- text avoids misevaluation cases, extremely com- mon in the random sentences-based evaluation se- tups.1 The higher IAA agreement in the random single-sentence setup is because raters tend to ac- cept the translation when adequacy is ambiguous but the translation is correct, especially if it is flu- ent. …

Content truncated to 500 words