Locally Distributed Activation Vectors for Guided Feature Attribution

Proceedings of the 29th International Conference on Computational Linguistics, pages 994–1005 October 12–17, 2022. 994 Locally Distributed Activation Vectors for Guided Feature Attribution Housam K. B. Babiker1, Mi-Young Kim2, Randy Goebel1 1 Department of Computing Science, University of Alberta 2 Department of Science, Augustana Faculty, University of Alberta Alberta Machine Intelligence Institute Edmonton, Alberta, Canada {khalifab, miyoung2, rgoebel}@ualberta.ca Abstract Explaining the predictions of a deep neural net- work (DNN) is a challenging problem. Many attempts at interpreting those predictions have focused on attribution-based methods, which assess the contributions of individual features to each model prediction. However, attribution- based explanations do not always provide faith- ful explanations to the target model, e.g., noisy gradients can result in unfaithful feature attribu- tion for back-propagation methods. We present a method to learn explanations-specific rep- resentations while constructing deep network models for text classification. These repre- sentations can be used to faithfully interpret black-box predictions, i.e., highlighting the most important input features and their role in any particular prediction. We show that learn- ing specific representations improves model interpretability across various tasks, for both qualitative and quantitative evaluations, while preserving predictive performance. 1 Introduction Deep neural network (DNN) models have become crucial tools in natural language processing (NLP) and define state-of-the-art on a large variety of tasks. However, DNN models are often consid- ered “black boxes,” whose predictions are difficult to interpret and understand. An immediate conse- quence is that quantifying the contribution of indi- vidual features is a challenging fundamental task in NLP and explainable AI research. In most related work, whether implicitly or explicitly, an explana- tion’s role in NLP text classification is to reveal which words and phrases are the most salient for the final prediction (Bastings and Filippova, 2020). From this viewpoint, a popular approach for ex- plaining a prediction is to use attribution methods, which justify the prediction of a pre-trained deep network, i.e., the explanation approximates the fea- ture attribution w.r.t. the predicted class. How- ever, attribution techniques, also included in the class of methods called post-hoc methods, might not always provide explanations that are faithful to the underlined model because of the instability of the explanations. This is largely because they often rely on heuristic rules; how those rules mimic the predictive calculation of the black box has the limitation of correlation with expected model be- havior (Rudin, 2018). For instance, (Alvarez-Melis and Jaakkola, 2018) showed that the explanations of two very close prediction points varied signifi- cantly in a simulated setting. We know that faithful explanations are essential, especially in high-stakes domains. If the explanations are wrong, we cannot trust the black box model. In addition, explana- tions are supposed to be faithful to what the model actually computes, so they may not meet the end user’s expectations. We define a faithful explana- tion in the context of NLP as follows: an explana- tion method is faithful if it is capable of identifying the most salient/meaningful features used by the model to make a prediction. The …

Content truncated to 500 words