On the Interaction of Belief Bias and Explanations

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2930–2942 August 1–6, 2021. ©2021 Association for Computational Linguistics 2930 On the Interaction of Belief Bias and Explanations Ana Valeria Gonz´alez1, Anna Rogers2, and Anders Søgaard1 University of Copenhagen 1Department of Computer Science 2Copenhagen Centre for Social Data Science {ana,soegaard}@di.ku.dk arogers@sodas.ku.dk Abstract A myriad of explainability methods have been proposed in recent years, but there is little con- sensus on how to evaluate them. While auto- matic metrics allow for quick benchmarking, it isn’t clear how such metrics reflect human interaction with explanations. Human eval- uation is of paramount importance, but pre- vious protocols fail to account for belief bi- ases affecting human performance, which may lead to misleading conclusions. We provide an overview of belief bias, its role in human evaluation, and ideas for NLP practitioners on how to account for it. For two experimental paradigms, we present a case study of gradient- based explainability introducing simple ways to account for humans’ prior beliefs: models of varying quality and adversarial examples. We show that conclusions about the highest performing methods change when introducing such controls, pointing to the importance of ac- counting for belief bias in evaluation. 1 Introduction Machine learning has become an integral part of our lives; from everyday use (e.g., search, transla- tion, recommendations) to high-stake applications in healthcare, law, or transportation. However, its impact is controversial: neural models have been shown to make confident predictions relying on arti- facts (McCoy et al., 2019; Wallace et al., 2019) and have shown to encode and amplify negative social biases (Manzini et al., 2019; Caliskan et al., 2017; May et al., 2019; Tan and Celis, 2019; Gonz´alez et al., 2020; Rudinger et al., 2018). Explainability aims to make model decisions transparent and predictable to humans; it serves as a tool for model diagnosis, detecting failure modes and biases, and more generally, to increase trust by providing transparency (Amershi et al., 2019). While automatic metrics have been proposed to Figure 1: Evaluation protocols considered in this work evaluate various properties of explanations such as faithfulness, consistency and agreement with hu- man explanations (Atanasova et al., 2020; Robnik- ˇSikonja and Bohanec, 2018; DeYoung et al., 2020), these metrics do not inform us about human inter- action with explanations. Doshi-Velez and Kim (2017) suggested human forward prediction, a simulation task in which hu- mans are given an input and an explanation, and their task is to predict the expected model out- put, regardless of the gold answer. Recent stud- ies include Nguyen (2018); Lage et al. (2019); ?); Poursabzi-Sangdeh et al. (2021). Such protocols are widely used and can provide valuable insight into human understanding of explanations. How- ever, prior work has not accounted for how humans’ prior beliefs (belief biases) interact with the evalua- tion; simulating model decisions becomes an easier task when the model being evaluated makes pre- dictions which align with human expectations. We argue that not considering belief bias in such pro- tocols may lead to misleading …

Content truncated to 500 words