Discretized Integrated Gradients for Explaining Language Models

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10285–10299 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 10285 Discretized Integrated Gradients for Explaining Language Models Soumya Sanyal University of Southern California soumyasa@usc.edu Xiang Ren University of Southern California xiangren@usc.edu Abstract As a prominent attribution-based explanation algorithm, Integrated Gradients (IG) is widely adopted due to its desirable explanation ax- ioms and the ease of gradient computation. It measures feature importance by averaging the model’s output gradient interpolated along a straight-line path in the input data space. How- ever, such straight-line interpolated points are not representative of text data due to the inher- ent discreteness of the word embedding space. This questions the faithfulness of the gradi- ents computed at the interpolated points and consequently, the quality of the generated ex- planations. Here we propose Discretized In- tegrated Gradients (DIG), which allows effec- tive attribution along non-linear interpolation paths. We develop two interpolation strategies for the discrete word embedding space that generates interpolation points that lie close to actual words in the embedding space, yield- ing more faithful gradient computation. We demonstrate the effectiveness of DIG over IG through experimental and human evaluations on multiple sentiment classification datasets. We provide the source code of DIG to encour- age reproducible research 1. 1 Introduction In the past few years, natural language processing has seen tremendous progress, largely due to strong performances yielded by pre-trained language mod- els (Devlin et al., 2019; Radford et al., 2019; Brown et al., 2020). But even with this impressive perfor- mance, it can still be difficult to understand the underlying reasoning for the preferred predictions leading to distrust among end-users (Lipton, 2018). Hence, improving model interpretability has be- come a central focus in the community with an increasing effort in developing methods that can ex- plain model behaviors (Ribeiro et al., 2016; Binder et al., 2016; Li et al., 2016; Sundararajan et al., 1https://github.com/INK-USC/DIG decent nice club okay smart home dull <pad> Input: the movie was good ! good Figure 1: An illustration of paths used in IG and DIG. IG uses a straight line interpolation with points as depicted by green squares. In contrast, DIG uses a non- linear path (shown in blue) with interpolation points (red stars) lying close to words in the embedding space. 2017; Shrikumar et al., 2017; Lundberg and Lee, 2017; Murdoch et al., 2018). Explanations in NLP are typically represented at a word-level or phrase-level by quantifying the con- tributions of the words or phrases to the model’s prediction by a scalar score. These explanation methods are commonly referred as attribution- based methods (Murdoch et al., 2018; Ancona et al., 2018). Integrated Gradients (IG) (Sundararajan et al., 2017) is a prominent attribution-based ex- planation method used due to the many desirable explanation axioms and ease of gradient compu- tation. It computes the partial derivatives of the model output with respect to each input feature as the features are interpolated along a straight-line path from the given input …

Content truncated to 500 words