Quantifying Attention Flow in Transformers

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4190–4197 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 4190 Quantifying Attention Flow in Transformers Samira Abnar ILLC, University of Amsterdam s.abnar@uva.nl Willem Zuidema ILLC, University of Amsterdam w.h.zuidema@uva.nl Abstract In the Transformer model, “self-attention” combines information from attended embed- dings into the representation of the focal em- bedding in the next layer. Thus, across lay- ers of the Transformer, information originating from different tokens gets increasingly mixed. This makes attention weights unreliable as ex- planations probes. In this paper, we consider the problem of quantifying this flow of infor- mation through self-attention. We propose two methods for approximating the attention to in- put tokens given attention weights, attention rollout and attention flow, as post hoc methods when we use attention weights as the relative relevance of the input tokens. We show that these methods give complementary views on the flow of information, and compared to raw attention, both yield higher correlations with importance scores of input tokens obtained us- ing an ablation method and input gradients. 1 Introduction Attention (Bahdanau et al., 2015; Vaswani et al., 2017) has become the key building block of neu- ral sequence processing models, and visualizing attention weights is the easiest and most popular approach to interpret a model’s decisions and to gain insights about its internals (Vaswani et al., 2017; Xu et al., 2015; Wang et al., 2016; Lee et al., 2017; Dehghani et al., 2019; Rockt¨aschel et al., 2016; Chen and Ji, 2019; Coenen et al., 2019; Clark et al., 2019). Although it is wrong to equate atten- tion with explanation (Pruthi et al., 2019; Jain and Wallace, 2019), it can offer plausible and mean- ingful interpretations (Wiegreffe and Pinter, 2019; Vashishth et al., 2019; Vig, 2019). In this paper, we focus on problems arising when we move to the higher layers of a model, due to lack of token iden- tifiability of the embeddings in higher layers (Brun- ner et al., 2020). We propose two simple but effective methods to compute attention scores to input tokens (i.e., token attention) at each layer, by taking raw attentions (i.e., embedding attention) of that layer as well as those from the precedent layers. These methods are based on modelling the information flow in the network with a DAG (Directed Acyclic Graph), in which the nodes are input tokens and hidden em- beddings, edges are the attentions from the nodes in each layer to those in the previous layer, and the weights of the edges are the attention weights. The first method, attention rollout, assumes that the identities of input tokens are linearly combined through the layers based on the attention weights. To adjust attention weights, it rolls out the weights to capture the propagation of information from in- put tokens to intermediate hidden embeddings. The second method, attention flow, considers the atten- tion graph as a flow network. Using a maximum flow algorithm, it computes maximum flow values, from …

Content truncated to 500 words