Local Interpretation of Transformer Based on Linear Decomposition

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 10270–10287 July 9-14, 2023 ©2023 Association for Computational Linguistics Local Interpretation of Transformer Based on Linear Decomposition Sen Yang, Shujian Huang∗, Wei Zou, Jianbing Zhang, Xinyu Dai, Jiajun Chen National Key Laboratory for Novel Software Technology, Nanjing University {yangsen,zouw}@smail.nju.edu.cn {huangsj,zjb,daixinyu,chenjj}@nju.edu.cn Abstract In recent years, deep neural networks (DNNs) have achieved state-of-the-art performance on a wide range of tasks. However, limitations in interpretability have hindered their applications in the real world. This work proposes to inter- pret neural networks by linear decomposition and finds that the ReLU-activated Transformer can be considered as a linear model on a sin- gle input. We further leverage the linearity of the model and propose a linear decomposition of the model output to generate local expla- nations. Our evaluation of sentiment classifi- cation and machine translation shows that our method achieves competitive performance in ef- ficiency and fidelity of explanation. In addition, we demonstrate the potential of our approach in applications with examples of error analysis on multiple tasks.1 1 Introduction Deep neural networks (DNNs) such as Trans- formers (Vaswani et al., 2017) have achieved state-of-the-art results on various natural language tasks (Devlin et al., 2019; Liu et al., 2019; Brown et al., 2020; Dai et al., 2019) via learning complex nonlinear relationships of inputs. However, the lack of interpretability of the pre- dictions given by black-box models limits their application in real-world (Guidotti et al., 2018; Lip- ton, 2018; Ribeiro et al., 2016b). A typical way to understand model prediction dynamics is to generate prediction explanations for each input, called local explanation genera- tion (Chen et al., 2020). Most existing works on local explanation algorithms in NLP strive to un- derstand such dynamics on word-level or phrase- level by assigning importance scores on input fea- tures (Ribeiro et al., 2016a; Lei et al., 2016; Lund- berg et al., 2018; Plumb et al., 2018). However, ∗ Corresponding author. 1We release our algorithm toolkit at https://github. com/DoubleVII/pydec. nonlinearity in models makes it difficult to assign the contribution of individual words or phrases to predictions, while the linear counterparts are more interpretable as the weight of each component could be naturally interpreted as its contribution. In this work, we present a linear decomposition theory to interpret linear models, which can be gen- eralized to nonlinear DNNs. That is, we formalize the decomposition of the linear outputs into com- ponents corresponding to the input features, then mathematically propose the properties of linear de- composition and the uniqueness of the decomposi- tion under these properties. Furthermore, we prove that the ReLU-activated Transformer can be regarded as a linear function by given input features if the causal relationship be- tween the input and certain intermediate variables is disregarded. Therefore, generalize the proposed linear decomposition to Transformer under such an assumption. However, this decomposition yields a compo- nent corresponding to the parameters of additive bias (usually used in the linear layer), which con- tains a …

Content truncated to 500 words