TANDO: A Corpus for Document-level Machine Translation

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 3026–3037 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 3026 TANDO: A Corpus for Document-level Machine Translation Harritxu Gete1∗, Thierry Etchegoyhen1∗, David Ponce1, Gorka Labaka2, Nora Aranberri2, Ander Corral3, Xabier Saralegi3, Igor Ellakuria Santos4, Maite Martin5 1Vicomtech Foundation, Basque Research and Technology Alliance (BRTA) 2IXA taldea, University of the Basque Country, 3Elhuyar, 4ISEA, 5Ametzagai˜na 1{hgete, tetchegoyhen, adponce}@vicomtech.org, 2{gorka.labaka, nora.aranberri}@ehu.eus, 3{x.saralegi, a.corral}@elhuyar.eus, 4iellakuria@isea.eus, 5maite@adur.com Abstract Document-level Neural Machine Translation aims to increase the quality of neural translation models by taking into account contextual information. Properly modelling information beyond the sentence level can result in improved machine translation output in terms of coherence, cohesion and consistency. Suitable corpora for context-level modelling are necessary to both train and evaluate context-aware systems, but are still relatively scarce. In this work we describe TANDO, a document-level corpus for the under-resourced Basque-Spanish language pair, which we share with the scientific community. The corpus is composed of parallel data from three different domains and has been prepared with context-level information. Additionally, the corpus includes contrastive test sets for fine-grained evaluations of gender and register contextual phenomena on both source and target language sides. To establish the usefulness of the corpus, we trained and evaluated baseline Transformer models and context-aware variants based on context concatenation. Our results indicate that the corpus is suitable for fine-grained evaluation of document-level machine translation systems. Keywords: Document-level Machine Translation, Parallel Corpus, Contrastive tests, Basque, Spanish 1. Introduction Neural machine translation (NMT) typically performs translation by considering sentences in isolation, ignor- ing discursive phenomena (Bahdanau et al., 2015; Lu- ong et al., 2015; Vaswani et al., 2017). Despite the improvements achieved at the sentence level, inconsis- tencies due to the lack of extra-sentential information raise important issues in terms of coherence, cohesion and consistency (L¨aubli et al., 2018; Toral et al., 2018; Voita et al., 2019b). Context-aware NMT is currently an active area of research and has notably been in- cluded among the shared tasks at the WMT conference series in recent years (Barrault et al., 2019; Barrault et al., 2020). Despite initial advances in context-aware models that tackle the generation of consistent and coherent trans- lations in context, advances in this field are hampered by two main issues. First, parallel corpora that include contextual information are relatively scarce (Liu and Zhang, 2020). Second, standard machine translation (MT) metrics, such as BLEU (Papineni et al., 2002), are usually not sensitive to improvements at the extra- sentential level (Wong and Kit, 2012; Sennrich, 2017). This work aims to address both of the aforementioned deficiencies. Regarding the first issue, we prepared a novel parallel dataset that includes contextual infor- mation for the Basque-Spanish language pair, cover- ing different domains such as literature, subtitles and varied news. To address the second issue, we pre- pared several contrastive test sets aimed at evaluating the ability of NMT models to handle common contex- tual errors in Basque-Spanish translation, namely gen- …

Content truncated to 500 words