Token-wise Decomposition of Autoregressive Language Model Hidden States for Analyzing Model Predictions

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 10105–10117 July 9-14, 2023 ©2023 Association for Computational Linguistics Token-wise Decomposition of Autoregressive Language Model Hidden States for Analyzing Model Predictions Byung-Doh Oh Department of Linguistics The Ohio State University oh.531@osu.edu William Schuler Department of Linguistics The Ohio State University schuler.77@osu.edu Abstract While there is much recent interest in studying why Transformer-based large language models make predictions the way they do, the complex computations performed within each layer have made their behavior somewhat opaque. To mit- igate this opacity, this work presents a linear decomposition of final hidden states from au- toregressive language models based on each ini- tial input token, which is exact for virtually all contemporary Transformer architectures. This decomposition allows the definition of proba- bility distributions that ablate the contribution of specific input tokens, which can be used to analyze their influence on model probabilities over a sequence of upcoming words with only one forward pass from the model. Using the change in next-word probability as a measure of importance, this work first examines which context words make the biggest contribution to language model predictions. Regression ex- periments suggest that Transformer-based lan- guage models rely primarily on collocational associations, followed by linguistic factors such as syntactic dependencies and coreference re- lationships in making next-word predictions. Additionally, analyses using these measures to predict syntactic dependencies and coreferent mention spans show that collocational associa- tion and repetitions of the same token largely explain the language models’ predictions on these tasks. 1 Introduction Much of contemporary natural language process- ing (NLP) is driven by Transformer-based large language models, which are trained to make predic- tions about words in their context by aggregating representations through their self-attention mecha- nism. The breakthrough in many NLP tasks these models have achieved has led to active research into interpreting their predictions and probing the knowledge embodied by these models (Manning et al., 2020; Rogers et al., 2021; Belinkov, 2022). Figure 1: Schematic of input and output representations from Transformer-based autoregressive language mod- els. Standard models (top) calculate one vector of final hidden states at a given timestep (xL,i), which in this work (bottom) is decomposed exactly into the sum of output representations of each input token (xL,i,k) and a cumulative bias term (bL,i). One line of such research focuses on quantifying the importance of each input token to the mod- els’ final output, but due to the complexity of the computations performed within the Transformer layers, analysis has been limited to studying the self-attention mechanism and the feedforward neu- ral network independently (Kobayashi et al., 2020, 2021; Geva et al., 2021, 2022; Mickus et al., 2022) or has relied on e.g. gradient-based attribution methods (Sanyal and Ren, 2021; Zaman and Be- linkov, 2022) that yield measures that are not inter- pretable in terms of output model probabilities. To address these limitations, this work presents a linear decomposition of final language model 10105 hidden states into the sum …

Content truncated to 500 words