When a Good Translation is Wrong in Context: Context-Aware Machine Translation Improves on Deixis, Ellipsis, and Lexical Cohesion

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1198–1212 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 1198 When a Good Translation is Wrong in Context: Context-Aware Machine Translation Improves on Deixis, Ellipsis, and Lexical Cohesion Elena Voita1,2 Rico Sennrich3,4 Ivan Titov3,2 1Yandex, Russia 2University of Amsterdam, Netherlands 3University of Edinburgh, Scotland 4University of Zurich, Switzerland lena-voita@yandex-team.ru rico.sennrich@ed.ac.uk ititov@inf.ed.ac.uk Abstract Though machine translation errors caused by the lack of context beyond one sentence have long been acknowledged, the development of context-aware NMT systems is hampered by several problems. Firstly, standard metrics are not sensitive to improvements in consistency in document-level translations. Secondly, pre- vious work on context-aware NMT assumed that the sentence-aligned parallel data con- sisted of complete documents while in most practical scenarios such document-level data constitutes only a fraction of the available par- allel data. To address the first issue, we per- form a human study on an English-Russian subtitles dataset and identify deixis, ellipsis and lexical cohesion as three main sources of inconsistency. We then create test sets target- ing these phenomena. To address the second shortcoming, we consider a set-up in which a much larger amount of sentence-level data is available compared to that aligned at the doc- ument level. We introduce a model that is suitable for this scenario and demonstrate ma- jor gains over a context-agnostic baseline on our new benchmarks without sacrificing per- formance as measured with BLEU.1 1 Introduction With the recent rapid progress of neural machine translation (NMT), translation mistakes and in- consistencies due to the lack of extra-sentential context are becoming more and more notice- able among otherwise adequate translations pro- duced by standard context-agnostic NMT systems (Läubli et al., 2018). Though this problem has recently triggered a lot of attention to context- aware translation (Jean et al., 2017a; Wang et al., 2017; Tiedemann and Scherrer, 2017; Bawden 1We release code and data sets at https://github.com/lena-voita/ good-translation-wrong-in-context. et al., 2018; Voita et al., 2018; Maruf and Haf- fari, 2018; Agrawal et al., 2018; Miculicich et al., 2018; Zhang et al., 2018), the progress and wide- spread adoption of the new paradigm is hampered by several important problems. Firstly, it is highly non-trivial to design metrics which would reliably trace the progress and guide model design. Stan- dard machine translation metrics (e.g., BLEU) do not appear appropriate as they do not sufficiently differentiate between consistent and inconsistent translations (Wong and Kit, 2012).2 For exam- ple, if multiple translations of a name are pos- sible, forcing consistency is essentially as likely to make all occurrences of the name match the reference translation as making them all different from the reference. Second, most previous work on context-aware NMT has made the assumption that all the bilingual data is available at the doc- ument level. However, isolated parallel sentences are a lot easier to acquire and hence only a frac- tion of the parallel data will be at the document level in any practical …

Content truncated to 500 words