Analyzing Transformers in Embedding Space

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 16124–16170 July 9-14, 2023 ©2023 Association for Computational Linguistics Analyzing Transformers in Embedding Space Guy Dar1 Mor Geva2 Ankit Gupta1 Jonathan Berant1 1The Blavatnik School of Computer Science, Tel-Aviv University 2Allen Institute for Artificial Intelligence {guy.dar,joberant}@cs.tau.ac.il, morp@allenai.org, ankitgupta.iitkanpur@gmail.com Abstract Understanding Transformer-based models has attracted significant attention, as they lie at the heart of recent technological advances across machine learning. While most interpretabil- ity methods rely on running models over in- puts, recent work has shown that an input- independent approach, where parameters are interpreted directly without a forward/back- ward pass is feasible for some Transformer pa- rameters, and for two-layer attention networks. In this work, we present a conceptual frame- work where all parameters of a trained Trans- former are interpreted by projecting them into the embedding space, that is, the space of vocabulary items they operate on. Focusing mostly on GPT-2 for this paper, we provide di- verse evidence to support our argument. First, an empirical analysis showing that parameters of both pretrained and fine-tuned models can be interpreted in embedding space. Second, we present two applications of our framework: (a) aligning the parameters of different mod- els that share a vocabulary, and (b) construct- ing a classifier without training by “translat- ing” the parameters of a fine-tuned classifier to parameters of a different model that was only pretrained. Overall, our findings show that at least in part, we can abstract away model specifics and understand Transformers in the embedding space. 1 Introduction Transformer-based models [Vaswani et al., 2017] cur- rently dominate Natural Language Processing [Devlin et al., 2018; Radford et al., 2019; Zhang et al., 2022] as well as many other fields of machine learning [Doso- vitskiy et al., 2020; Chen et al., 2020; Baevski et al., 2020]. Consequently, understanding their inner work- ings has been a topic of great interest. Typically, work on interpreting Transformers relies on feeding inputs to the model and analyzing the resulting activations [Adi et al., 2016; Shi et al., 2016; Clark et al., 2019]. Thus, interpretation involves an expensive forward, and sometimes also a backward pass, over multiple inputs. Moreover, such interpretation methods are conditioned on the input and are not guaranteed to generalize to all inputs. In the evolving literature on static interpreta- tion, i.e., without forward or backward passes, [Geva et al., 2022b] showed that the value vectors of the Transformer feed-forward module (the second layer of the feed-forward network) can be interpreted by pro- jecting them into the embedding space, i.e., multiply- ing them by the embedding matrix to obtain a repre- sentation over vocabulary items.1 [Elhage et al., 2021] have shown that in a 2-layer attention network, weight matrices can be interpreted in the embedding space as well. Unfortunately, their innovative technique could not be extended any further. In this work, we extend and unify the theory and findings of [Elhage et al., 2021] and [Geva et al., 2022b]. We present …

Content truncated to 500 words