Attention is not Explanation

Proceedings of NAACL-HLT 2019, pages 3543–3556 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 3543 Attention is not Explanation Sarthak Jain Northeastern University jain.sar@husky.neu.edu Byron C. Wallace Northeastern University b.wallace@northeastern.edu Abstract Attention mechanisms have seen wide adop- tion in neural NLP models. In addition to improving predictive performance, these are often touted as affording transparency: mod- els equipped with attention provide a distribu- tion over attended-to input units, and this is often presented (at least implicitly) as com- municating the relative importance of inputs. However, it is unclear what relationship ex- ists between attention weights and model out- puts. In this work we perform extensive exper- iments across a variety of NLP tasks that aim to assess the degree to which attention weights provide meaningful “explanations" for predic- tions. We find that they largely do not. For example, learned attention weights are fre- quently uncorrelated with gradient-based mea- sures of feature importance, and one can iden- tify very different attention distributions that nonetheless yield equivalent predictions. Our findings show that standard attention mod- ules do not provide meaningful explanations and should not be treated as though they do. Code to reproduce all experiments is avail- able at https://github.com/successar/ AttentionExplanation. 1 Introduction and Motivation Attention mechanisms (Bahdanau et al., 2014) in- duce conditional distributions over input units to compose a weighted context vector for down- stream modules. These are now a near-ubiquitous component of neural NLP architectures. Attention weights are often claimed (implicitly or explic- itly) to afford insights into the “inner-workings” of models: for a given output one can inspect the inputs to which the model assigned large attention weights. Li et al. (2016) summarized this com- monly held view in NLP: “Attention provides an important way to explain the workings of neural models". Indeed, claims that attention provides after 15 minutes watching the movie i was asking myself what to do leave the theater sleep or try to keep watching the movie to see if there was anything worth i finally watched the movie what a waste of time maybe i am not a 5 years old kid anymore original adversarial after 15 minutes watching the movie i was asking myself what to do leave the theater sleep or try to keep watching the movie to see if there was anything worth i finally watched the movie what a waste of time maybe i am not a 5 years old kid anymore f(x|↵, ✓) = 0.01 f(x|˜↵, ✓) = 0.01 ↵ ˜↵ Figure 1: Heatmap of attention weights induced over a negative movie review. We show observed model at- tention (left) and an adversarially constructed set of at- tention weights (right). Despite being quite dissimilar, these both yield effectively the same prediction (0.01). interpretability are common in the literature, e.g., (Xu et al., 2015; Choi et al., 2016; Lei et al., 2017; Martins and Astudillo, 2016; Xie et al., 2017).1 Implicit in this is the assumption that the input units (e.g., words) accorded high …

Content truncated to 500 words