Effective Attention Sheds Light On Interpretability

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4126–4135 August 1–6, 2021. ©2021 Association for Computational Linguistics 4126 Effective Attention Sheds Light On Interpretability Kaiser Sun* Ana Marasovi´c†* ∗Paul G. Allen School of Computer Science & Engineering, University of Washington †Allen Institute for AI Seattle, WA, USA huikas@cs.washington.edu, anam@allenai.org Abstract An attention matrix of a transformer self- attention sublayer can provably be decom- posed into two components and only one of them (effective attention) contributes to the model output. This leads us to ask whether visualizing effective attention gives different conclusions than interpretation of standard at- tention. Using a subset of the GLUE tasks and BERT, we carry out an analysis to compare the two attention matrices, and show that their in- terpretations differ. Effective attention is less associated with the features related to the lan- guage modeling pretraining such as the separa- tor token, and it has more potential to illustrate linguistic features captured by the model for solving the end-task. Given the found differ- ences, we recommend using effective attention for studying a transformer’s behavior since it is more pertinent to the model output by design. 1 Introduction Attention mechanism (Bahdanau et al., 2015) is an essential component of many NLP models, in- cluding those that are built on the ubiquitous trans- former architecture (Vaswani et al., 2017). As a result, visualizing attention weights is a widely used technique to interpret models’ behavior (Be- linkov and Glass, 2019). Despite that, the validity of this analysis method is a subject undergoing intense discussion and study in NLP (Jain and Wal- lace, 2019; Wiegreffe and Pinter, 2019; Serrano and Smith, 2019; Moradi et al., 2019; Mohanku- mar et al., 2020; Tutek and Snajder, 2020, i.a.). Related to this discussion, Brunner et al. (2020) show that, under mild conditions, the attention ma- trix of a transformer self-attention sublayer can be written as a sum of two components. One of them is irrelevant for the model output because its product with the value matrix is zero. They term the other component as effective attention (formally defined in §2). We study whether effective attention gives interpretations that differ from conclusions we get by analyzing standard attention. If this is the case, interpretation of effective attention is better suited for studying transformers’ internals because it is more pertinent to the model output by design. Brunner et al. (2020) briefly discuss this by com- paring standard and effective attention matrices from a single BERT head (Devlin et al., 2019) for one example. They observe that: (i) standard atten- tion is largely concentrated on the delimiter tokens ([SEP], [CLS]) or on near-diagonal elements; (ii) effective attention is more dispersed; (iii) effective attention disregards the delimiters. They stress that we should not extrapolate too much from these ob- servations since they are based on a single example, and that further research is needed on this topic. In this work, we aim to reliably answer whether effective attention disregards the [SEP] and [CLS] tokens, and if …

Content truncated to 500 words