Quantifying Context Mixing in Transformers

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 3378–3400 May 2-6, 2023 ©2023 Association for Computational Linguistics Quantifying Context Mixing in Transformers Hosein Mohebbi1 Willem Zuidema2 Grzegorz Chrupała1 Afra Alishahi1 1 CSAI, Tilburg University 2 ILLC, University of Amsterdam {h.mohebbi, a.alishahi}@tilburguniversity.edu w.h.zuidema@uva.nl grzegorz@chrupala.me Abstract Self-attention weights and their transformed variants have been the main source of informa- tion for analyzing token-to-token interactions in Transformer-based models. But despite their ease of interpretation, these weights are not faithful to the models’ decisions as they are only one part of an encoder, and other compo- nents in the encoder layer can have consider- able impact on information mixing in the out- put representations. In this work, by expand- ing the scope of analysis to the whole encoder block, we propose Value Zeroing, a novel con- text mixing score customized for Transformers that provides us with a deeper understanding of how information is mixed at each encoder layer. We demonstrate the superiority of our context mixing score over other analysis meth- ods through a series of complementary evalu- ations with different viewpoints based on lin- guistically informed rationales, probing, and faithfulness analysis.1 1 Introduction Transformers (Vaswani et al., 2017), with their im- pressive empirical success, have become a prime choice of architecture to learn contextualized repre- sentations across a wide range of modalities, such as language (Devlin et al., 2019; Brown et al., 2020), vision (Dosovitskiy et al., 2021), vision- language (Radford et al., 2021; Rombach et al., 2022), and speech (Baevski et al., 2020), mainly due to their ability to utilize pairwise interactions between input tokens at every timestep. To better understand the inner dynamics of Transformers, we need to trace the information flow from the input embeddings up to the output representation (including quantifying the degree of context mixing, which we will define below). The attention weights from the multi-head atten- tion mechanisms offer a straightforward starting 1Code is freely available at https://github.com/ hmohebbi/ValueZeroing point for understanding this flow, and these weights (‘raw attention’) have been used in many studies (Clark et al., 2019; Kovaleva et al., 2019; Reif et al., 2019; Htut et al., 2019a, inter alia). However, the reliability and usefulness of raw attention weights has also been questioned (Jain and Wallace, 2019; Bibal et al., 2022). In particular, attention weights tend to concentrate on uninformative tokens in the context (Voita et al., 2018; Clark et al., 2019), and removing or altering them may lead to the same and sometimes even better model performance on downstream tasks (Jain and Wallace, 2019; Toneva and Wehbe, 2019; Hassid et al., 2022). These find- ings suggest that Transformers do not solely rely on self-attention, and other components in the encoder block play an essential role in information mixing. A number of methods have been proposed to compute some form of ‘effective attention weights’, with the goal of more faithfully tracing the relative contributions of different input tokens at various layers of the Transformer (as we …

Content truncated to 500 words