A Large-Scale Test Set for the Evaluation of Context-Aware Pronoun Translation in Neural Machine Translation

Proceedings of the Third Conference on Machine Translation (WMT), Volume 1: Research Papers, pages 61–72 Belgium, Brussels, October 31 - Novermber 1, 2018. c⃝2018 Association for Computational Linguistics https://doi.org/10.18653/v1/W18-64007 A Large-Scale Test Set for the Evaluation of Context-Aware Pronoun Translation in Neural Machine Translation Mathias Müller1,2 Annette Rios1 Elena Voita3,4 Rico Sennrich1,5 1Institute of Computational Linguistics, University of Zurich 2Amazon Research, Berlin∗ 3Yandex Research, Russia 4University of Amsterdam, Netherlands 5School of Informatics, University of Edinburgh Abstract The translation of pronouns presents a special challenge to machine translation to this day, since it often requires con- text outside the current sentence. Re- cent work on models that have access to information across sentence boundaries has seen only moderate improvements in terms of automatic evaluation metrics such as BLEU. However, metrics that quan- tify the overall translation quality are ill- equipped to measure gains from additional context. We argue that a different kind of evaluation is needed to assess how well models translate inter-sentential phenom- ena such as pronouns. This paper therefore presents a test suite of contrastive transla- tions focused specifically on the transla- tion of pronouns. Furthermore, we per- form experiments with several context- aware models. We show that, while gains in BLEU are moderate for those systems, they outperform baselines by a large mar- gin in terms of accuracy on our contrastive test set. Our experiments also show the ef- fectiveness of parameter tying for multi- encoder architectures. 1 Introduction Even though machine translation has improved considerably with the advent of neural machine translation (NMT) (Sutskever et al., 2014; Bah- danau et al., 2015), the translation of pronouns re- mains a major issue. They are notoriously hard to translate since they often require context outside the current sentence. As an example, consider the sentences in Fig- ure 1. In both languages, there is a pronoun in the ∗ Work performed prior to joining Amazon. EN However, the European Central Bank (ECB) took an interest in it. It describes bitcoin as “the most successful virtual currency”. DE Dennoch hat die Europäische Zentralbank (EZB) Interesse hierfür gezeigt. Sie beschreibt Bitcoin als “die virtuelle Währung mit dem grössten Erfolg”. Figure 1: Example sentence illustrating how the translation of pronouns is ambiguous on a sen- tence level. Pronouns of interest are in italics, and the antecedents they refer to are underlined. Taken from WMT newstest2013. second sentence that refers to the European Cen- tral Bank. When the second sentence is translated from English to German, the translation of the pro- noun it is ambiguous. This ambiguity can only be resolved with context awareness: if a transla- tion system has access to the previous English sen- tence, the previous German translation, or both, it can determine the antecedent the pronoun refers to. In this German sentence, the antecedent Eu- ropäische Zentralbank dictates the feminine gen- der of the pronoun sie. It is unfortunate, then, that current NMT sys- tems generally operate on the sentence level (Vaswani et al., 2017; Gehring et al., 2017; Hieber …

Content truncated to 500 words