Inserting Information Bottlenecks for Attribution in Transformers

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3850–3857 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 3850 Inserting Information Bottlenecks for Attribution in Transformers Zhiying Jiang, Raphael Tang, Ji Xin and Jimmy Lin David R. Cheriton School of Computer Science University of Waterloo {zhiying.jiang, r33tang, ji.xin, jimmylin}@uwaterloo.ca Abstract Pretrained transformers achieve the state of the art across tasks in natural language processing, motivating researchers to investigate their in- ner mechanisms. One common direction is to understand what features are important for pre- diction. In this paper, we apply information bottlenecks to analyze the attribution of each feature for prediction on a black-box model. We use BERT as the example and evaluate our approach both quantitatively and qualitatively. We show the effectiveness of our method in terms of attribution and the ability to provide insight into how information flows through lay- ers. We demonstrate that our technique out- performs two competitive methods in degrada- tion tests on four datasets. Code is available at https://github.com/bazingagin/IBA. 1 Introduction Increasingly prominent is the urge to interpret deep neural networks, with the success of these black- box models remaining vastly inexplicable both the- oretically and empirically. Within natural language processing (NLP), this desire is particularly true for the pretrained transformer, which has witnessed an influx of literature on interpretability analysis. Such papers include visualizing transformer atten- tion mechanisms (Kovaleva et al., 2019), probing the geometry of transformer representations (He- witt and Manning, 2019), and explaining the span predictions of question answering models (van Aken et al., 2019). In this paper, we focus on prediction attribution methods. That is, we ask, “Which hidden features contribute the most toward a prediction?” To re- solve this question, a number of methods (Selvaraju et al., 2017; Smilkov et al., 2017) generate attribu- tion scores for features, which provide a human- understandable “explanation” of how a particular prediction is made at the instance level. Specif- ically, given an instance, these methods assign a numerical score for each hidden feature denoting its relevance toward the prediction. Previous papers have demonstrated that gradient- based methods fail to capture all the information as- sociated with the correct prediction (Li et al., 2016). To address this weakness, Schulz et al. (2020) insert information bottlenecks (Tishby et al., 2000) for attribution, attaining both stronger empirical per- formance and a theoretical upper bound on the in- formation used. Additionally, mutual information is unconstrained by model and task (Guan et al., 2019). Thus, we adopt information bottlenecks for attribution (IBA) to interpret transformer models at the instance level. We apply IBA to BERT (Devlin et al., 2019) across five datasets in sentiment analy- sis, textual entailment, and document classification. We show both qualitatively and quantitatively that the method capably captures information in the model’s token-level features, as well as insight into cross-layer behavior. Our contributions are as follows: First, we are the first to apply information bottlenecks (IB) for attri- bution to explain transformers. Second, we conduct quantitative analysis to investigate the …

Content truncated to 500 words