Leakage-Adjusted Simulatability: Can Models Generate Non-Trivial Explanations of Their Behavior in Natural Language?

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4351–4367 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 4351 Leakage-Adjusted Simulatability: Can Models Generate Non-Trivial Explanations of Their Behavior in Natural Language? Peter Hase, Shiyue Zhang, Harry Xie, and Mohit Bansal UNC Chapel Hill {peter, shiyue, fengyu.xie, mbansal}@cs.unc.edu Abstract Data collection for natural language (NL) un- derstanding tasks has increasingly included hu- man explanations alongside data points, allow- ing past works to introduce models that both perform a task and generate NL explanations for their outputs. Yet to date, model-generated explanations have been evaluated on the ba- sis of surface-level similarities to human expla- nations, both through automatic metrics like BLEU and human evaluations. We argue that these evaluations are insufficient, since they fail to indicate whether explanations support actual model behavior (faithfulness), rather than simply match what a human would say (plausibility). In this work, we address the problem of evaluating explanations from the model simulatability perspective. Our contri- butions are as follows: (1) We introduce a leakage-adjusted simulatability (LAS) metric for evaluating NL explanations, which mea- sures how well explanations help an observer predict a model’s output, while controlling for how explanations can directly leak the out- put. We use a model as a proxy for a hu- man observer, and validate this choice with two human subject experiments. (2) Using the CoS-E and e-SNLI datasets, we evaluate two existing generative graphical models and two new approaches; one rationalizing method we introduce achieves roughly human-level LAS scores. (3) Lastly, we frame explanation gen- eration as a multi-agent game and optimize explanations for simulatability while penaliz- ing label leakage, which can improve LAS scores.1 1 Introduction Deep neural models have achieved impressive suc- cess in many areas. However, their interpretability 1We provide code for the experiments in this paper at https://github.com/peterbhase/ LAS-NL-Explanations. and explainability have remained broadly limited. To make neural models more interpretable, previ- ous works have proposed methods for explaining model decisions, e.g., through various feature im- portance estimates (Hendricks et al., 2018; Ribeiro et al., 2016) or model-generated natural language (NL) (Hendricks et al., 2016; Kim et al., 2018). Early work on generating NL explanations focused on providing explanations that were both descrip- tive of an image and discriminative as labels (Hen- dricks et al., 2016). Since then, a variety of datasets have been collected with free-form human gener- ated explanations accompanying each data point (Camburu et al., 2018; Kim et al., 2018; Zellers et al., 2019; Wang et al., 2019; Rajani et al., 2019). Models have been proposed for these datasets with two aims: (1) to teach models how to explain their own decisions in natural language, by offering demonstrations of humans doing this, and (2) to in- crease model accuracy on the task, by making use of additional information in human explanations. Past works have proposed varying methods for generating NL explanations, which can be repre- sented by distinct graphical models. In our work, we explore four graphical models, shown in …

Content truncated to 500 words