On Context Span Needed for Machine Translation Evaluation

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 3735–3742 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 3735 On Context Span Needed for Machine Translation Evaluation Sheila Castilho, Maja Popovi´c, Andy Way ADAPT Centre, School of Computing Dublin City University {sheila.castilho, maja.popovic, andy.way}@adaptcentre.ie Abstract Despite increasing efforts to improve evaluation of machine translation (MT) by going beyond the sentence level to the document level, the definition of what exactly constitutes a “document level” is still not clear. This work deals with the context span necessary for a more reliable MT evaluation. We report results from a series of surveys involving three domains and 18 target languages designed to identify the necessary context span as well as issues related to it. Our findings indicate that, despite the fact that some issues and spans are strongly dependent on domain and on the target language, a number of common patterns can be observed so that general guidelines for context-aware MT evaluation can be drawn. Keywords: MT evaluation, document-level MT evaluation, human evaluation 1. Introduction One of the biggest challenges for MT is the ability to handle discourse dependencies and the wider context of a docu- ment. Although currently an active community is working on developing discourse-level MT systems, the improve- ments reported for those systems are still limited. One of the main reasons for this is that the evaluation of document- level systems (both automatic and human) has primarily been performed at the sentence level and is, therefore, un- able to recognise the real improvements of document-level systems. Furthermore, two recent studies (Toral et al., 2018; L¨aubli et al., 2018) independently reassessed the bold claims of MT “achieving human parity” (Hassan et al., 2018) and found that the lack of extra-sentential context has a great effect on quality assessment. After these two papers were published, WMT19 considered their criticisms and at- tempted, for the first time, a document-level human evalua- tion for some language pairs. Despite the increased in the field, both for extending MT systems to operate on the document level, as well as for im- proving the evaluation methodology by expanding it to the document level, the definition of what exactly constitutes a “document level” evaluation is still not clear. At this point, it is uncertain whether it refers to pairs of consecutive sen- tences, to a paragraph, or even to whole chapter. Our paper tests the context span necessary for human evaluation on three different domains, guided by the two research ques- tions: • RQ1 - Are two previous sentences enough for reliable evaluation? • RQ2 - What are the linguistic issues related to the nec- essary context span? Despite certain limitations of our experiments (small cor- pora, unbalanced number of participants for different target languages and domains, as well as lack of specific guide- lines), we believe that our findings represent a good starting point towards reliable context-aware MT evaluation. In the following section (section 2.), we present the …

Content truncated to 500 words