An Analysis of Attention Mechanisms: The Case of Word Sense Disambiguation in Neural Machine Translation

Proceedings of the Third Conference on Machine Translation (WMT), Volume 1: Research Papers, pages 26–35 Belgium, Brussels, October 31 - Novermber 1, 2018. c⃝2018 Association for Computational Linguistics https://doi.org/10.18653/v1/W18-64004 An Analysis of Attention Mechanisms: The Case of Word Sense Disambiguation in Neural Machine Translation Gongbo Tang1 Rico Sennrich2,3 Joakim Nivre1 1Department of Linguistics and Philology, Uppsala University 2School of Informatics, University of Edinburgh 3Institute of Computational Linguistics, University of Zurich firstname.lastname@{lingfil.uu.se, ed.ac.uk} Abstract Recent work has shown that the encoder- decoder attention mechanisms in neural ma- chine translation (NMT) are different from the word alignment in statistical machine trans- lation. In this paper, we focus on analyz- ing encoder-decoder attention mechanisms, in the case of word sense disambiguation (WSD) in NMT models. We hypothesize that atten- tion mechanisms pay more attention to context tokens when translating ambiguous words. We explore the attention distribution patterns when translating ambiguous nouns. Counter- intuitively, we find that attention mechanisms are likely to distribute more attention to the ambiguous noun itself rather than context to- kens, in comparison to other nouns. We con- clude that attention is not the main mecha- nism used by NMT models to incorporate con- textual information for WSD. The experimen- tal results suggest that NMT models learn to encode contextual information necessary for WSD in the encoder hidden states. For the at- tention mechanism in Transformer models, we reveal that the first few layers gradually learn to “align” source and target tokens and the last few layers learn to extract features from the re- lated but unaligned context tokens. 1 Introduction Human languages exhibit many different types of ambiguity. Lexical ambiguity refers to the fact that words can have more than one semantic meaning. Dealing with these lexical ambiguities is a chal- lenge for various NLP tasks. Word sense disam- biguation (WSD) is recognizing the correct mean- ing of an ambiguous word, with the help of con- textual information. In statistical machine translation (SMT) (Koehn et al., 2003), a system could explicitly take context tokens into account to improve the translation of ambiguous words (Vickrey et al., 2005). By con- trast, in neural machine translation (NMT) (Kalch- brenner and Blunsom, 2013; Sutskever et al., 2014; Cho et al., 2014), especially in attentional NMT (Bahdanau et al., 2015; Luong et al., 2015), each hidden state incorporates contextual informa- tion. Hence, NMT models could potentially per- form WSD well. However, there are no empiri- cal results to indicate that the hidden states encode the contextual information needed for disambigua- tion. Moreover, how the attention mechanism1 deals with ambiguous words is also not known yet. In this paper, we focus on the question of how encoder-decoder attention mechanisms deal with ambiguous nouns. We explore two different atten- tion mechanisms. One is the vanilla one-layer at- tention mechanism (Bahdanau et al., 2015; Luong et al., 2015), and the other one is the Transformer attention mechanism (Vaswani et al., 2017). Rios et al. (2017) find that attentional NMT models perform well in translating ambiguous words …

Content truncated to 500 words