DELA Corpus - A Document-Level Corpus Annotated with Context-Related Issues

Proceedings of the Sixth Conference on Machine Translation (WMT), pages 566–577 November 10–11, 2021. ©2021 Association for Computational Linguistics 566 DELA Corpus - A Document-Level Corpus Annotated with Context-Related Issues Sheila Castilho ADAPT Centre - School of Computing Dublin City University sheila.castilho@adaptcentre.ie João L. Cavalheiro Camargo Western Paraná State University joao.camargo@unioeste.br Miguel Menezes University of Lisbon lmenezes@campus.ul.pt Andy Way ADAPT Centre - School of Computing Dublin City University andy.way@adaptcentre.ie Abstract Recently, the Machine Translation (MT) community has become more interested in document-level evaluation especially in light of reactions to claims of "human parity", since examining the quality at the level of the doc- ument rather than at the sentence level allows for the assessment of suprasentential context, providing a more reliable evaluation. This paper presents a document-level corpus an- notated in English with context-aware issues that arise when translating from English into Brazilian Portuguese, namely ellipsis, gender, lexical ambiguity, number, reference, and ter- minology, with six different domains. The cor- pus can be used as a challenge test set for eval- uation and as a training/testing corpus for MT as well as for deep linguistic analysis of con- text issues. To the best of our knowledge, this is the first corpus of its kind. 1 Introduction Machine translation (MT) is now widely used in a variety of fields, mainly due to advancements in neural models (Sutskever et al., 2014; Bahdanau et al., 2015; Vaswani et al., 2017). As a result of these recent advances, scientists have been in- creasingly attempting to include discourse into neural machine translation (NMT) systems (Wang, 2019; Lopes et al., 2020). Thus, researchers started to consider a more suitable evaluation for these document-level systems as the standard MT auto- matic evaluation metrics have been shown to un- derestimate the quality of NMT systems (Shteri- onov et al., 2018) and the appropriateness of these metrics for document-level systems has been chal- lenged (Smith, 2017) since they are not sensitive to their improvements (Voita et al., 2019). Accordingly, document-level human evaluation of MT has attracted the community’s attention since it allows for a more thorough examination of the output quality with context. While a few works have taken into account document-level hu- man evaluation (Läubli et al., 2018; Toral et al., 2018; Barrault et al., 2019; Castilho, 2020, 2021), one common practice for document-level evalua- tion is the usage of test suites with context-aware markers (Bawden et al., 2018; Guillou et al., 2018; Müller et al., 2018; Voita et al., 2019; Cai and Xiong, 2020). However, the concept of document- level evaluation, in terms of how much text must be shown, remains uncertain (Castilho et al., 2020). While most research on document-level MT evalu- ation works with contrastive pairs, very few works have tried to use full documents for human evalua- tion (Läubli et al., 2018; Castilho, 2020, 2021) and challenge test sets (Rysová et al., 2019; Vojtˇechová et al., 2019). Methodologies for assessing MT at the document-level have been looked into (Barrault et al., 2019, 2020) as …

Content truncated to 500 words