What does Attention in Neural Machine Translation Pay Attention to?

Proceedings of the The 8th International Joint Conference on Natural Language Processing, pages 30–39, Taipei, Taiwan, November 27 – December 1, 2017 c⃝2017 AFNLP What does Attention in Neural Machine Translation Pay Attention to? Hamidreza Ghader and Christof Monz Informatics Institute, University of Amsterdam, The Netherlands h.ghader, c.monz@uva.nl Abstract Attention in neural machine translation provides the possibility to encode relevant parts of the source sentence at each trans- lation step. As a result, attention is con- sidered to be an alignment model as well. However, there is no work that specifically studies attention and provides analysis of what is being learned by attention mod- els. Thus, the question still remains that how attention is similar or different from the traditional alignment. In this paper, we provide detailed analysis of attention and compare it to traditional alignment. We answer the question of whether attention is only capable of modelling translational equivalent or it captures more information. We show that attention is different from alignment in some cases and is capturing useful information other than alignments. 1 Introduction Neural machine translation (NMT) has gained a lot of attention recently due to its substantial im- provements in machine translation quality achiev- ing state-of-the-art performance for several lan- guages (Luong et al., 2015b; Jean et al., 2015; Wu et al., 2016). The core architecture of neural machine translation models is based on the gen- eral encoder-decoder approach (Sutskever et al., 2014). Neural machine translation is an end-to- end approach that learns to encode source sen- tences into distributed representations and decode these representations into sentences in the target language. Among the different neural MT models, attentional NMT (Bahdanau et al., 2015; Luong et al., 2015a) has become popular due to its capa- bility to use the most relevant parts of the source sentence at each translation step. This capability also makes the attentional model superior in trans- lating longer sentences (Bahdanau et al., 2015; Lu- ong et al., 2015a). Figure 1: Visualization of the attention paid to the relevant parts of the source sentence for each gen- erated word of a translation example. See how the attention is ‘smeared out’ over multiple source words in the case of “would” and “like”. Figure 1 shows an example of how attention uses the most relevant source words to generate a target word at each step of the translation. In this paper we focus on studying the relevance of the attended parts, especially cases where atten- tion is ‘smeared out’ over multiple source words where their relevance is not entirely obvious, see, e.g., “would” and “like” in Figure 1. Here, we ask whether these are due to errors of the attention mechanism or are a desired behavior of the model. Since the introduction of attention models in neural machine translation (Bahdanau et al., 2015) various modifications have been proposed (Lu- ong et al., 2015a; Cohn et al., 2016; Liu et al., 2016). However, to the best of our knowledge 30 there is no study that provides an …

Content truncated to 500 words