Integrated Directional Gradients: Feature Interaction Attribution for Neural NLP Models

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 865–878 August 1–6, 2021. ©2021 Association for Computational Linguistics 865 Integrated Directional Gradients: Feature Interaction Attribution for Neural NLP Models Sandipan Sikdar∗ RWTH Aachen University sandipan.sikdar@ cssh.rwth-aachen.de Parantapa Bhattacharya∗ University of Virginia parantapa@ virginia.edu Kieran Heese University of Virginia kh8fb@ virginia.edu Abstract In this paper, we introduce Integrated Direc- tional Gradients (IDG), a method for attribut- ing importance scores to groups of features, indicating their relevance to the output of a neural network model for a given input. The success of Deep Neural Networks has been at- tributed to their ability to capture higher level feature interactions. Hence, in the last few years capturing the importance of these fea- ture interactions has received increased promi- nence in ML interpretability literature. In this paper, we formally define the feature group attribution problem and outline a set of ax- ioms that any intuitive feature group attribu- tion method should satisfy. Earlier, cooper- ative game theory inspired axiomatic meth- ods only borrowed axioms from solution con- cepts (such as Shapley value) for individual feature attributions and introduced their own extensions to model interactions. In contrast, our formulation is inspired by axioms satis- fied by characteristic functions as well as solu- tion concepts in cooperative game theory liter- ature. We believe that characteristic functions are much better suited to model importance of groups compared to just solution concepts. We demonstrate that our proposed method, IDG, satisfies all the axioms. Using IDG we an- alyze two state-of-the-art text classifiers on three benchmark datasets for sentiment analy- sis. Our experiments show that IDG is able to effectively capture semantic interactions in lin- guistic models via negations and conjunctions. 1 Introduction In the last decade Deep Neural Networks (DNN) have been immensely successful. Much of this success can be attributed to their ability to learn from complex higher order interactions from raw features (Goodfellow et al., 2016). This success of DNNs has led to them being increasingly adopted ∗Equal contribution for algorithmic decision making. This in turn has led to increasing concerns over explainabil- ity and interpretability of these models, given the important role they are beginning to take in soci- ety (Selbst and Barocas, 2018). One area of work that has emerged in recent years is that of black box model explanation strate- gies that “explain” the output of a DNN for a given input using feature attribution scores or saliency maps (Sundararajan et al., 2017; Shrikumar et al., 2017). Numerous studies have been published in recent years proposing different strategies to an- swer the question “which features in the input were most important in deciding the output of the DNN?” However, modern DNNs take as input raw data as features, and learn from higher order interaction of those features. Thus in the past year a number of studies have instead focused on explaining fea- ture interactions rather than explaining individual features (Chen …

Content truncated to 500 words