Evaluating Pronominal Anaphora in Machine Translation: An Evaluation Measure and a Test Suite

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 2964–2975, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 2964 Evaluating Pronominal Anaphora in Machine Translation: An Evaluation Measure and a Test Suite Prathyusha Jwalapuram∗, Shafiq Joty∗§, Irina Temnikova†, and Preslav Nakov‡ ∗Nanyang Technological University, Singapore §Salesforce Research Asia, Singapore †Sofia University, Bulgaria ‡Qatar Computing Research Institute, Doha, Qatar ∗{jwal0001,srjoty}@ntu.edu.sg †irina.temnikova@gmail.com ‡pnakov@hbku.edu.qa Abstract The neural revolution in machine translation has made it easier to model larger contexts beyond the sentence-level, which can poten- tially help resolve some discourse-level ambi- guities such as pronominal anaphora, thus en- abling better translations. Unfortunately, even when the resulting improvements are seen as substantial by humans, they remain virtually unnoticed by traditional automatic evaluation measures such as BLEU, as only a few words end up being affected. Thus, specialized eval- uation measures are needed. With this aim in mind, we contribute an extensive, targeted dataset that can be used as a test suite for pronoun translation, covering multiple source languages and different pronoun errors drawn from real system translations, for English. We further propose an evaluation measure to dif- ferentiate good and bad pronoun translations. Finally, we conduct a user study and we report correlation with human judgments. 1 Introduction Traditionally, machine translation (MT) has been performed at the level of individual sentences, i.e., in isolation from the rest of the document. This was due to the nature of the underlying frameworks: word-based (Brown et al., 1993), then phrase-based (Koehn et al., 2003), syntac- tic (Galley et al., 2004), and hierarchical (Chiang, 2005). While there have been attempts to model the context beyond the sentence level, e.g., looking at neighboring sentences (Carpuat and Wu, 2007; Chan et al., 2007) or even at the entire document (Hardmeier et al., 2012), these approaches were still limited by the underlying framework. Then, along came the neural revolution. Thanks to the attention mechanism, neural translation models such as sequence-to-sequence (Bahdanau et al., 2015) and the Transformer (Vaswani et al., 2017) could model much broader context. While initially translation was still done in a sentence-by-sentence fashion, researchers soon re- alized that going beyond that has become easier, and recent work has successfully exploited this (Bawden et al., 2018; Voita et al., 2018). This is an exciting research direction as it can help address inter-sentential phenomena such as anaphora, gen- der agreement, lexical consistency, and text coher- ence, to mention just a few. Unfortunately, going beyond the sentence level typically yields very few changes in the translation output, and even when these changes are seen as substantial by humans, they remain virtually un- noticed by typical MT evaluation measures such as BLEU (Papineni et al., 2002), which are known to be notoriously problematic for the evaluation of discourse-level aspects in MT (Hardmeier, 2014). The limitations of BLEU are well-known and have been discussed in detail in a recent study (Re- iter, 2018). It has long …

Content truncated to 500 words