Evaluating Explanations: How Much Do Explanations from the Teacher Aid Students?

Evaluating Explanations: How Much Do Explanations from the Teacher Aid Students? Danish Pruthi1∗ Rachit Bansal2 Bhuwan Dhingra3 Livio Baldini Soares3 Michael Collins3 Zachary C. Lipton1 Graham Neubig1 William W. Cohen3 1 Carnegie Mellon University, USA 2 Delhi Technological University, India 3 Google Research, USA {ddanish, zlipton, gneubig}@cs.cmu.edu, racbansa@gmail.com {bdhingra, liviobs, mjcollins, wcohen}@google.com Abstract While many methods purport to explain pre- dictions by highlighting salient features, what aims these explanations serve and how they ought to be evaluated often go unstated. In this work, we introduce a framework to quan- tify the value of explanations via the accuracy gains that they confer on a student model trained to simulate a teacher model. Cru- cially, the explanations are available to the student during training, but are not available at test time. Compared with prior proposals, our approach is less easily gamed, enabling principled, automatic, model-agnostic evalu- ation of attributions. Using our framework, we compare numerous attribution methods for text classification and question answer- ing, and observe quantitative differences that are consistent (to a moderate to high degree) across different student model architectures and learning strategies.1 1 Introduction The success of deep learning models, together with the difficulty of understanding how they work, has inspired a subfield of research on explaining predictions, often by highlighting specific input features deemed somehow important to a pre- diction (Ribeiro et al., 2016; Sundararajan et al., 2017; Shrikumar et al., 2017). For instance, we might expect such a method to highlight spans like ‘‘poorly acted’’ and ‘‘slow-moving’’ to explain a prediction of negative sentiment for a given movie review. However, there is little agreement in the literature as to what constitutes a good explana- ∗Part of this work was done at Google. 1Code for the evaluation protocol: https://github .com/danishpruthi/evaluating-explanations. tion (Lipton, 2016; Jacovi and Goldberg, 2021). Moreover, various popular methods for generat- ing such attributions disagree considerably over which tokens to highlight (Table 1). With so many methods claimed to confer the same property while disagreeing so markedly, one path forward is to develop clear quantitative criteria for evaluating purported explanations at scale. The status quo for evaluating so-called expla- nations skews qualitative—many proposed tech- niques are evaluated only via visual inspection of a few examples (Simonyan et al., 2014; Sundararajan et al., 2017; Shrikumar et al., 2017). While several quantitative evaluation tech- niques have recently been proposed, many of these are easily gamed (Treviso and Martins, 2020; Hase et al., 2020).2 Some depend upon the model outputs corresponding to deformed exam- ples that lie outside the support of the training distribution (DeYoung et al., 2020), and a few validate explanations on specifically crafted tasks (Poerner et al., 2018). In this work, we propose a new framework, where explanations are quantified by the degree to which they help a student model in learn- ing to simulate the teacher on future examples (Figure 1). Our framework addresses a coherent goal, is model-agnostic and broadly applicable across tasks, and (when instantiated with models as students) can easily be automated …

Content truncated to 500 words