Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5540–5552 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 5540 Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior? Peter Hase and Mohit Bansal UNC Chapel Hill peter@cs.unc.edu, mbansal@cs.unc.edu Abstract Algorithmic approaches to interpreting ma- chine learning models have proliferated in re- cent years. We carry out human subject tests that are the first of their kind to isolate the ef- fect of algorithmic explanations on a key as- pect of model interpretability, simulatability, while avoiding important confounding experi- mental factors. A model is simulatable when a person can predict its behavior on new in- puts. Through two kinds of simulation tests in- volving text and tabular data, we evaluate five explanations methods: (1) LIME, (2) Anchor, (3) Decision Boundary, (4) a Prototype model, and (5) a Composite approach that combines explanations from each method. Clear ev- idence of method effectiveness is found in very few cases: LIME improves simulatabil- ity in tabular classification, and our Prototype method is effective in counterfactual simula- tion tests. We also collect subjective ratings of explanations, but we do not find that rat- ings are predictive of how helpful explanations are. Our results provide the first reliable and comprehensive estimates of how explanations influence simulatability across a variety of ex- planation methods and data domains. We show that (1) we need to be careful about the metrics we use to evaluate explanation methods, and (2) there is significant room for improvement in current methods.1 1 Introduction Interpretable machine learning is now a widely discussed topic (Rudin, 2019; Doshi-Velez and Kim, 2017; Lipton, 2016; Gilpin et al., 2018). While survey papers have not converged on def- initions of “explainable” or “interpretable,” there are some common threads in the discourse. Com- mentators observe that interpretability is useful for 1We make all our supporting code, data, and mod- els publicly available at: https://github.com/peterbhase/ InterpretableNLP-ACL2020 achieving other model desiderata, which may in- clude building user trust, identifying the influence of certain variables, understanding how a model will behave on given inputs, and ensuring that models are fair and unbiased. In their review, Doshi-Velez and Kim (2017) outline an approach to measuring interpretability. They describe two human-subject tasks that test for a particularly useful property: simulatability. A model is simulatable when a person can predict its behavior on new inputs. This property is espe- cially useful since it indicates that a person under- stands why a model produces the outputs it does. The first of the two tasks is termed forward simu- lation: given an input and an “explanation,” users must predict what a model would output for the given input. The second is counterfactual simula- tion: users are given an input, a model’s output for that input, and an “explanation” of that output, and then they must predict what the model will out- put when given a perturbation of the original in- put. The explanation itself is algorithmically gen- erated by …

Content truncated to 500 words