Is Attention Explanation? An Introduction to the Debate

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 3889 - 3900 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Is Attention Explanation? An Introduction to the Debate Adrien Bibal, Rémi Cardon, David Alfter, Rodrigo Wilkens, Xiaoou Wang, Thomas François∗and Patrick Watrin∗ CENTAL, IL&C, University of Louvain, Belgium {adrien.bibal,remi.cardon,david.alfter,xiaoou.wang, thomas.francois,patrick.watrin}@uclouvain.be Abstract The performance of deep learning models in NLP and other fields of machine learning has led to a rise in their popularity, and so the need for explanations of these models be- comes paramount. Attention has been seen as a solution to increase performance, while providing some explanations. However, a de- bate has started to cast doubt on the explana- tory power of attention in neural networks. Al- though the debate has created a vast literature thanks to contributions from various areas, the lack of communication is becoming more and more tangible. In this paper, we provide a clear overview of the insights on the debate by crit- ically confronting works from these different areas. This holistic vision can be of great in- terest for future works in all the communities concerned by this debate. We sum up the main challenges spotted in these areas, and we con- clude by discussing the most promising future avenues on attention as an explanation. 1 Introduction Attention mechanisms have been widely used in various tasks of Natural Language Processing (NLP) as well as in other fields of machine learning (e.g., Computer Vision (Mnih et al., 2014; Li et al., 2019)). These mechanisms draw insight from the intuition that humans build the representation of a whole scene by dynamically focusing on relevant parts at different times (Rensink, 2000). The general form of attention has been named differently according to authors (alignment model (Bahdanau et al., 2015) and attention mech- anism (Vaswani et al., 2017)). In essence, the at- tention function maps a query Q and keys K to scalar scores (Vaswani et al., 2017). These scores are fed to a softmax function, in turn producing a set of attention weights that are then applied to values V. Different kinds of attention are thus pos- sible according to how many keys are attended to ∗T. François and P. Watrin are co-last authors. (global vs. local attention, according to Luong et al. (2015)) and where the query is generated (cross vs. self-attention as in the works of Bah- danau et al. (2015) and Vaswani et al. (2017)). In this paper, we focus on attention regardless of these technical differences. There are mainly two ways of computing the attention weights ˆα: Bah- danau et al. (2015) introduced additive attention ˆα = softmax(w3Ttanh(W1K + W2Q)), where w3, W1, W2 model parameters to be learned, and Vaswani et al. (2017) introduced scaled dot-product attention ˆα = softmax � KQ √m � , where m represents the dimension of K. These two forms are theoreti- cally similar (Vaswani et al., 2017) and generally give the same results (Jain and Wallace, 2019), …

Content truncated to 500 words