Attention is not not Explanation

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 11–20, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 11 Attention is not not Explanation Sarah Wiegreffe⇤ School of Interactive Computing Georgia Institute of Technology saw@gatech.edu Yuval Pinter⇤ School of Interactive Computing Georgia Institute of Technology uvp@gatech.edu Abstract Attention mechanisms play a central role in NLP systems, especially within recurrent neu- ral network (RNN) models. Recently, there has been increasing interest in whether or not the intermediate representations offered by these modules may be used to explain the rea- soning for a model’s prediction, and conse- quently reach insights regarding the model’s decision-making process. A recent paper claims that ‘Attention is not Explanation’ (Jain and Wallace, 2019). We challenge many of the assumptions underlying this work, argu- ing that such a claim depends on one’s defi- nition of explanation, and that testing it needs to take into account all elements of the model. We propose four alternative tests to determine when/whether attention can be used as ex- planation: a simple uniform-weights baseline; a variance calibration based on multiple ran- dom seed runs; a diagnostic framework using frozen weights from pretrained models; and an end-to-end adversarial attention training pro- tocol. Each allows for meaningful interpreta- tion of attention mechanisms in RNN models. We show that even when reliable adversarial distributions can be found, they don’t perform well on the simple diagnostic, indicating that prior work does not disprove the usefulness of attention mechanisms for explainability. 1 Introduction Attention mechanisms (Bahdanau et al., 2014) are nowadays ubiquitous in NLP, and their suitabil- ity for providing explanations for model predic- tions is a topic of high interest (Xu et al., 2015; Rockt¨aschel et al., 2015; Mullenbach et al., 2018; Thorne et al., 2019; Serrano and Smith, 2019). If they indeed offer such insights, many applica- tion areas would benefit by better understanding the internals of neural models that use attention ⇤Equal contributions. as a means for, e.g., model debugging or architec- ture selection. A recent paper (Jain and Wallace, 2019) points to possible pitfalls that may cause re- searchers to misapply attention scores as explana- tions of model behavior, based on a premise that explainable attention distributions should be con- sistent with other feature-importance measures as well as exclusive given a prediction.1 Its core ar- gument, which we elaborate in §2, is that if al- ternative attention distributions exist that produce similar results to those obtained by the original model, then the original model’s attention scores cannot be reliably used to “faithfully” explain the model’s prediction. Empirically, the authors show that achieving such alternative distributions is easy for a large sample of English-language datasets. We contend (§2.1) that while Jain and Wal- lace ask an important question, and raise a gen- uine concern regarding potential misuse of atten- tion weights in explaining model decisions on English-language datasets, some key assumptions used in their experimental design leave an …

Content truncated to 500 words