Generalizing Backpropagation for Gradient-Based Interpretability

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 11979–11995 July 9-14, 2023 ©2023 Association for Computational Linguistics Generalizing Backpropagation for Gradient-Based Interpretability Kevin DuD Lucas Torroba Hennigen@ Niklas StoehrD Alexander WarstadtD Ryan CotterellD DETH Zürich @MIT kevin.du@inf.ethz.ch lucastor@mit.edu niklas.stoehr@inf.ethz.ch alexanderscott.warstadt@inf.ethz.ch ryan.cotterell@inf.ethz.ch Abstract Many popular feature-attribution methods for interpreting deep neural networks rely on com- puting the gradients of a model’s output with respect to its inputs. While these methods can indicate which input features may be impor- tant for the model’s prediction, they reveal lit- tle about the inner workings of the model it- self. In this paper, we observe that the gradient computation of a model is a special case of a more general formulation using semirings. This observation allows us to generalize the backpropagation algorithm to efficiently com- pute other interpretable statistics about the gra- dient graph of a neural network, such as the highest-weighted path and entropy. We imple- ment this generalized algorithm, evaluate it on synthetic datasets to better understand the statis- tics it computes, and apply it to study BERT’s behavior on the subject–verb number agree- ment task (SVA). With this method, we (a) val- idate that the amount of gradient flow through a component of a model reflects its importance to a prediction and (b) for SVA, identify which pathways of the self-attention mechanism are most important. 1 Introduction1 One of the key contributors to the success of deep learning in NLP has been backpropagation (Lin- nainmaa, 1976), a dynamic programming algo- rithm that efficiently computes the gradients of a scalar function with respect to its inputs (Goodfel- low et al., 2016). Backpropagation works by con- structing a directed acyclic computation graph2 that describes a function as a composition of various primitive operations, e.g., +, ×, and exp(·), whose gradients are known, and subsequently traversing this graph in topological order to incrementally compute the gradients. Since the runtime of back- propagation is linear in the number of edges of 1Code and data available at https://github.com/ kdu4108/semiring-backprop-exps. 2With due care, a computation graph can be extended to the cyclic case. the computation graph, it is possible to quickly per- form vast numbers of gradient descent steps in even the most gargantuan of neural networks. While gradients are arguably most important for training, they can also be used to analyze and interpret neural network behavior. For example, feature attribution methods such as saliency maps (Simonyan et al., 2013) and integrated gradi- ents (Sundararajan et al., 2017) exploit gradients to identify which features of an input contribute most towards the model’s prediction. However, most of these methods provide little insight into how the gradient propagates through the computation graph, and those that do are computationally inefficient, e.g., Lu et al. (2021) give an algorithm for computing the highest-weighted gradient path that runs in exponential time. In this paper, we explore whether examining var- ious quantities computed from the gradient graph of a network, i.e., the weighted graph whose …

Content truncated to 500 words