Encoders Help You Disambiguate Word Senses in Neural Machine Translation

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 1429–1435, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 1429 Encoders Help You Disambiguate Word Senses in Neural Machine Translation Gongbo Tang1 Rico Sennrich2,3 Joakim Nivre1 1Department of Linguistics and Philology, Uppsala University 2Institute of Computational Linguistics, University of Zurich 3School of Informatics, University of Edinburgh firstname.lastname@{lingfil.uu.se, ed.ac.uk} Abstract Neural machine translation (NMT) has achieved new state-of-the-art performance in translating ambiguous words. However, it is still unclear which component dominates the process of disambiguation. In this paper, we explore the ability of NMT encoders and decoders to disambiguate word senses by evaluating hidden states and investigating the distributions of self-attention. We train a classifier to predict whether a translation is correct given the representation of an ambigu- ous noun. We find that encoder hidden states outperform word embeddings significantly which indicates that encoders adequately en- code relevant information for disambiguation into hidden states. In contrast to encoders, the effect of decoder is different in models with different architectures. Moreover, the attention weights and attention entropy show that self-attention can detect ambiguous nouns and distribute more attention to the context. 1 Introduction Neural machine translation (NMT) models (Kalchbrenner and Blunsom, 2013; Sutskever et al., 2014; Cho et al., 2014; Bahdanau et al., 2015; Luong et al., 2015) have access to the whole source sentence for the prediction of each word, which intuitively allows them to perform word sense disambiguation (WSD) better than previous phrase-based methods, and Rios et al. (2018) have confirmed this empirically. However, it is still unclear which component dominates the ability to disambiguate word senses. We explore the ability of NMT encoders and decoders to disambiguate word senses by evaluating hidden states and investigating the self-attention distributions. Marvin and Koehn (2018) find that the hidden states in higher encoder layers do not perform dis- ambiguation better than those in lower layers and conclude that encoders do not encode enough rel- evant context for disambiguation. However, their results are based on small data sets, and we wish to revisit this question with larger-scale data sets. Tang et al. (2018b) speculate that encoders have encoded the relevant information for WSD into hidden states before decoding but without any ex- perimental tests. In this paper, we first train a classifier for WSD, on a much larger data set than Marvin and Koehn (2018), extracted from ContraWSD (Rios et al., 2017), for both German→English (DE→EN) and German→French (DE→FR). The classifier is fed a representation of ambiguous nouns and a word sense (represented as the embedding of a trans- lation candidate), and has to predict whether the two match. We can learn the role that encoders play in encoding information relevant for WSD by comparing different representations: word embed- dings and encoder hidden states at different layers. We extract encoder hidden states from both RNN- based (RNNS2S) (Luong et al., 2015) and Trans- former (Vaswani et al., …

Content truncated to 500 words