Do Context-Aware Translation Models Pay the Right Attention?

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 788–801 August 1–6, 2021. ©2021 Association for Computational Linguistics 788 Do Context-Aware Translation Models Pay the Right Attention? Kayo Yin1 Patrick Fernandes1,2,3 Danish Pruthi1 Aditi Chaudhary1 Andr´e F. T. Martins2,3,4 Graham Neubig1 1Language Technologies Institute, Carnegie Mellon University, Pittsburgh, PA 2Instituto Superior T´ecnico & LUMLIS (Lisbon ELLIS Unit), Lisbon, Portugal 3Instituto de Telecomunicac¸˜oes, Lisbon, Portugal 4Unbabel, Lisbon, Portugal {kayoy, pfernand, ddanish, aschaudh, gneubig}@cs.cmu.edu andre.t.martins@tecnico.ulisboa.pt Abstract Context-aware machine translation models are designed to leverage contextual information, but often fail to do so. As a result, they in- accurately disambiguate pronouns and polyse- mous words that require context for resolu- tion. In this paper, we ask several questions: What contexts do human translators use to re- solve ambiguous words? Are models paying large amounts of attention to the same context? What if we explicitly train them to do so? To answer these questions, we introduce SCAT (Supporting Context for Ambiguous Transla- tions), a new English-French dataset compris- ing supporting context words for 14K trans- lations that professional translators found use- ful for pronoun disambiguation. Using SCAT, we perform an in-depth analysis of the context used to disambiguate, examining positional and lexical characteristics of the supporting words. Furthermore, we measure the degree of alignment between the model’s attention scores and the supporting context from SCAT, and apply a guided attention strategy to encour- age agreement between the two.1 1 Introduction There is a growing consensus in machine trans- lation research that it is necessary to move be- yond sentence-level translation and incorporate document-level context (Guillou et al., 2018; L¨aubli et al., 2018; Toral et al., 2018). While various methods to incorporate context in neural machine translation (NMT) have been proposed (Tiedemann and Scherrer (2017); Miculicich et al. (2018); Maruf and Haffari (2018), inter alia), it is unclear whether models rely on the “right” con- text that is actually sufficient to disambiguate dif- ficult translations. Even when additional context 1Our SCAT data and code for experiments are available at https://github.com/neulab/contextual-mt. Human En Look after her a lot. Okay. Any questions? Have we got her report? Yes, it’s in the infirmary already Fr Dorlotez-la. D’accord. Vous avez des questions ? On dispose de son rapport. Oui, il est `a l’infirmerie. Context-aware baseline En Look after her a lot. Okay. Any questions? Have we got her report? Yes, it’s in the infirmary already. Fr Dorlotez-la. D’accord. Vous avez des questions ? On dispose de son rapport ? Oui, elle est d´ej`a `a l’infirmerie. Model w/ attention regularization En Look after her a lot. Okay. Any questions? Have we got her report? Yes it’s in the infirmary already. Fr Dorlotez-la. D’accord. Vous avez des questions ? On dispose de son rapport ? Oui, il est d´ej`a `a l’hˆopital Table 1: Translation of the ambiguous pronoun “it”. In French, if the referent of “it” is masculine (e.g., report) then “il” is used, otherwise “elle”. The …

Content truncated to 500 words