Towards Transparent and Explainable Attention Models

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4206–4216 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 4206 Towards Transparent and Explainable Attention Models Akash Kumar Mohankumar1 Preksha Nema1,2 Sharan Narasimhan1 Mitesh M. Khapra1,2 Balaji Vasan Srinivasan3 Balaraman Ravindran1,2 1Indian Institute of Technology Madras 3Adobe Research 2 Robert Bosch Center for Data Science and Artificial Intelligence, IIT Madras makashkumar99@gmail.com {preksha,miteshk,ravi}@cse.iitm.ac.in sharan.n21@gmail.com balsrini@adobe.com Abstract Recent studies on interpretability of attention distributions have led to notions of faithful and plausible explanations for a model’s predic- tions. Attention distributions can be consid- ered a faithful explanation if a higher atten- tion weight implies a greater impact on the model’s prediction. They can be considered a plausible explanation if they provide a human- understandable justification for the model’s predictions. In this work, we first explain why current attention mechanisms in LSTM based encoders can neither provide a faithful nor a plausible explanation of the model’s predic- tions. We observe that in LSTM based en- coders the hidden representations at different time-steps are very similar to each other (high conicity) and attention weights in these situa- tions do not carry much meaning because even a random permutation of the attention weights does not affect the model’s predictions. Based on experiments on a wide variety of tasks and datasets, we observe attention distributions of- ten attribute the model’s predictions to unim- portant words such as punctuation and fail to offer a plausible explanation for the predic- tions. To make attention mechanisms more faithful and plausible, we propose a modified LSTM cell with a diversity-driven training ob- jective that ensures that the hidden represen- tations learned at different time steps are di- verse. We show that the resulting attention distributions offer more transparency as they (i) provide a more precise importance rank- ing of the hidden states (ii) are better indica- tive of words important for the model’s predic- tions (iii) correlate better with gradient-based attribution methods. Human evaluations indi- cate that the attention distributions learned by our model offer a plausible explanation of the model’s predictions. Our code has been made publicly available at https://github.com/ akashkm99/Interpretable-Attention 1 Introduction Question 1: What is the best way to improve my spoken English soon ? Question 2: How can I improve my English speaking ability ? Is paraphrase (Actual & Predicted): Yes Attention Distribution Vanilla LSTM How can I improve my English speaking ability ? Diversity LSTM How can I improve my English speaking ability ? Passage: Sandra went to the garden . Daniel went to the garden. Question: Where is Sandra? Answer (Actual & Predicted): garden Attention Distribution: Vanilla LSTM Sandra went to the garden . Daniel went to the garden Diversity LSTM Sandra went to the garden . Daniel went to the garden Table 1: Samples of Attention distributions from Vanilla and Diversity LSTM models on the Quora Question Paraphrase (QQP) & Babi 1 datasets. . Attention mechanisms (Bahdanau et al., 2014; Vaswani et al., 2017) play a very important …

Content truncated to 500 words