Connecting Attributions and QA Model Behavior on Realistic Counterfactuals

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5496–5512 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 5496 Connecting Attributions and QA Model Behavior on Realistic Counterfactuals Xi Ye Rohan Nair Greg Durrett Department of Computer Science The University of Texas at Austin {xiye,rnair,gdurrett}@cs.utexas.edu Abstract When a model attribution technique highlights a particular part of the input, a user might un- derstand this highlight as making a statement about counterfactuals (Miller, 2019): if that part of the input were to change, the model’s prediction might change as well. This paper investigates how well different attribution tech- niques align with this assumption on realistic counterfactuals in the case of reading compre- hension (RC). RC is a particularly challeng- ing test case, as token-level attributions that have been extensively studied in other NLP tasks such as sentiment analysis are less suit- able to represent the reasoning that RC models perform. We construct counterfactual sets for three different RC settings, and through heuris- tics that can connect attribution methods’ out- puts to high-level model behavior, we can eval- uate how useful different attribution methods and even different formats are for understand- ing counterfactuals. We find that pairwise at- tributions are better suited to RC than token- level attributions across these different RC set- tings, with our best performance coming from a modification that we propose to an existing pairwise attribution method.1 1 Introduction Recent research in interpretability of neural mod- els (Lipton, 2018) has yielded numerous post-hoc explanation methods, including token attribution techniques (Ribeiro et al., 2016; Sundararajan et al., 2017; Guan et al., 2019; De Cao et al., 2020). At- tributions are a flexible explanation format and can be applied to many domains, including sen- timent analysis (Guan et al., 2019; De Cao et al., 2020), visual recognition (Simonyan et al., 2013), and natural language inference (Camburu et al., 2018; Thorne et al., 2019). However, it is hard to evaluate whether these explanations are faithful 1Code and data: https://github.com/xiye17/EvalQAExpl Base Example Attributions ? Counterfactuals Expected Behavior Hypothesis My QA model is looking at these tokens simulate Given the explanation, we expect predictions to change in this way compare Actual Behavior run model Realistic counterfactual examples with those tokens changed Mismatch between explanations and model behavior on these counterfactuals. The explanation is bad! Figure 1: Our methodology. Given a base example, we can formulate a hypothesis about the model’s behav- ior, like a theory about how the model is using certain tokens. Next, we collect counterfactual examples that modify these tokens and profile the actual model be- havior. Finally, we assess whether feature attributions suggest behavior consistent with what we observe, veri- fying whether our attributions actually enable meaning- ful statements about behavior on counterfactuals. to the computation of the original model (Wu and Mooney, 2019; Hase and Bansal, 2020; Wiegreffe et al., 2021; Jacovi and Goldberg, 2020) and as a result, they can potentially mislead users (Rudin, 2019). Furthermore, attributions do not have a con- sistent and …

Content truncated to 500 words