Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5553–5563 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 5553 Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions Xiaochuang Han Carnegie Mellon University xiaochuh@cs.cmu.edu Byron C. Wallace Northeastern University b.wallace@northeastern.edu Yulia Tsvetkov Carnegie Mellon University ytsvetko@cs.cmu.edu Abstract Modern deep learning models for NLP are no- toriously opaque. This has motivated the de- velopment of methods for interpreting such models, e.g., via gradient-based saliency maps or the visualization of attention weights. Such approaches aim to provide explanations for a particular model prediction by highlighting im- portant words in the corresponding input text. While this might be useful for tasks where de- cisions are explicitly influenced by individual tokens in the input, we suspect that such high- lighting is not always suitable for tasks where model decisions should be driven by more complex reasoning. In this work, we inves- tigate the use of influence functions for NLP, providing an alternative approach to interpret- ing neural text classifiers. Influence functions explain the decisions of a model by identify- ing influential training examples. Despite the promise of this approach, influence functions have not yet been extensively evaluated in the context of NLP, a gap addressed by this work. We conduct a comparison between influence functions and common word-saliency methods on representative tasks. As suspected, we find that influence functions are particularly useful for natural language inference, a task in which ‘saliency maps’ may not provide clear interpre- tation. Furthermore, we develop a new quan- titative measure based on influence functions that can reveal artifacts in training data.1 1 Introduction Deep learning models have become increasingly complex, and unfortunately their inscrutability has grown in tandem with their predictive power (Doshi-Velez and Kim, 2017). This has motivated efforts to design example-specific approaches to interpreting black box NLP model predictions, i.e., 1Code is available at https://github.com/ xhan77/influence-function-analysis. indicating specific input tokens as being particu- larly influential for a given prediction. This in turn facilitates the construction of saliency maps over texts, in which words are highlighted with intensity proportional to continuous ‘importance’ scores. Prominent examples of the latter include gradient-based attribution (Simonyan et al., 2014; Sundararajan et al., 2017; Smilkov et al., 2017), LIME (Ribeiro et al., 2016), and attention-based (Xu et al., 2015) heatmaps. While widely used and potentially useful for some lexicon-driven tasks (e.g., sentiment analy- sis), we argue that by virtue of being constrained to highlighting individual input tokens, saliency maps will necessarily fail to explain predictions in more complex semantic tasks involving reasoning, such as natural language inference (NLI), where fine-grained interactions between multiple words or spans are key (Camburu et al., 2018). Moreover, saliency maps are inherently limited as a model debugging tool; they may tell us which inputs the model found to be important, but not why. To address these shortcomings, we investigate the use of what Lipton (2018) referred to as ex- planation by example. Instead of constructing im- portance scores over …

Content truncated to 500 words