Document-level Neural MT: A Systematic Comparison

Document-level Neural MT: A Systematic Comparison Ant´onio V. Lopes1 M. Amin Farajian1 Rachel Bawden2 Michael Zhang3 Andr´e F. T. Martins1 1Unbabel, Rua Visc. de Santar´em 67B, Lisbon, Portugal 2University of Edinburgh, Scotland, UK 3 University of Washington, Seattle, WA, USA {antonio.lopes, amin, andre.martins}@unbabel.com rachel.bawden@ed.ac.uk, mjqzhang@cs.washington.edu Abstract In this paper we provide a systematic com- parison of existing and new document- level neural machine translation solutions. As part of this comparison, we introduce and evaluate a document-level variant of the recently proposed Star Transformer ar- chitecture. In addition to using the tradi- tional metric BLEU, we report the accu- racy of the models in handling anaphoric pronoun translation as well as coherence and cohesion using contrastive test sets. Finally, we report the results of human evaluation in terms of Multidimensional Quality Metrics (MQM) and analyse the correlation of the results obtained by the automatic metrics with human judgments. 1 Introduction There has been undeniable progress in Machine Translation (MT) in recent years, so much so that for certain languages and domains, when sentences are evaluated in isolation, it has been suggested that MT is on par with human translation (Has- san et al., 2018). However, it has been shown that human translation clearly outperforms MT at the document level, when the whole translation is taken into account (L¨aubli et al., 2018; Toral et al., 2018; Laubli et al., 2020). For example, the Con- ference on Machine Translation (WMT) now con- siders inter-sentential translations in their shared task (Barrault et al., 2019). This sets a demand for context-aware machine translation: systems that take the context into account when translating, as opposed to translating sentences independently. © 2020 The authors. This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CC- BY-ND. Translating sentences in context (i.e. at the doc- ument level) is essential for correctly handling discourse phenomena whose scope can go be- yond the current sentence and which therefore re- quire document context (Hardmeier, 2012; Baw- den, 2018; Wang, 2019). Important examples in- clude anaphora, lexical coherence and cohesion, deixis and ellipsis; crucial aspects in delivering high quality translations which often are poorly evaluated using standard automatic metrics. Numerous context-aware neural MT (NMT) approaches have been proposed in recent years (Tiedemann and Scherrer, 2017; Zhang et al., 2018; Maruf et al., 2019; Miculicich et al., 2018; Voita et al., 2019b; Tu et al., 2018), integrat- ing source-side and sometimes target-side context. However, they have often been evaluated on differ- ent languages, datasets, and model sizes. Certain models have also previously been trained on few sentence pairs rather than in more realistic, high- resource scenarios. A direct comparison and anal- ysis of the methods, particularly concerning their individual strengths and weaknesses on different language pairs is therefore currently lacking. We fill these gaps by comparing a representa- tive set of context-aware NMT solutions under the same experimental settings, providing: • A systematic comparison of context-aware NMT methods using large datasets (i.e. pre-trained using large amounts of sentence-level data) …

Content truncated to 500 words