Is Attention Interpretable?

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2931–2951 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 2931 Is Attention Interpretable? Sofia Serrano∗ Noah A. Smith∗† ∗Paul G. Allen School of Computer Science & Engineering, University of Washington, Seattle, WA, USA †Allen Institute for Artificial Intelligence, Seattle, WA, USA {sofias6,nasmith}@cs.washington.edu Abstract Attention mechanisms have recently boosted performance on a range of NLP tasks. Because attention layers explicitly weight input compo- nents’ representations, it is also often assumed that attention can be used to identify informa- tion that models found important (e.g., specific contextualized word tokens). We test whether that assumption holds by manipulating atten- tion weights in already-trained text classifica- tion models and analyzing the resulting differ- ences in their predictions. While we observe some ways in which higher attention weights correlate with greater impact on model predic- tions, we also find many ways in which this does not hold, i.e., where gradient-based rank- ings of attention weights better predict their ef- fects than their magnitudes. We conclude that while attention noisily predicts input compo- nents’ overall importance to a model, it is by no means a fail-safe indicator.1 1 Introduction Interpretability is a pressing concern for many current NLP models. As they become increas- ingly complex and learn decision-making func- tions from data, ensuring our ability to understand why a particular decision occurred is critical. Part of that development has been the incorpo- ration of attention mechanisms (Bahdanau et al., 2015) into models for a variety of tasks. For many different problems—to name a few, ma- chine translation (Luong et al., 2015), syntactic parsing (Vinyals et al., 2015), reading comprehen- sion (Hermann et al., 2015), and language model- ing (Liu and Lapata, 2018)—incorporating atten- tion mechanisms into models has proven benefi- cial for performance. While there are many vari- ants of attention (Vaswani et al., 2017), each for- 1Code is available at https://github.com/ serrano-s/attn-tests. mulation consists of the same high-level goal: cal- culating nonnegative weights for each input com- ponent (e.g., word) that together sum to 1, multi- plying those weights by their corresponding repre- sentations, and summing the resulting vectors into a single fixed-length representation. Since attention calculates a distribution over in- puts, prior work has used attention as a tool for in- terpretation of model decisions (Wang et al., 2016; Lee et al., 2017; Lin et al., 2017; Ghaeini et al., 2018). The existence of so much work on visual- izing attention weights is a testament to attention’s popularity in this regard; to name just a few ex- amples of these weights being examined to under- stand a model, recent work has focused on goals from explaining and debugging the current sys- tem’s decision (Lee et al., 2017; Ding et al., 2017) to distilling important traits of a dataset (Yang et al., 2017; Habernal et al., 2018). Despite this, existing work on interpretability is only beginning to assess what computed atten- tion weights actually communicate. In an …

Content truncated to 500 words