Logic Traps in Evaluating Attribution Scores

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 5911 - 5922 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Logic Traps in Evaluating Attribution Scores Yiming Ju1,2, Yuanzhe Zhang1,2, Zhao Yang1,2, Zhongtao Jiang1,2, Kang Liu1,2,3, Jun Zhao1,2, 1 National Laboratory of Pattern Recognition, Institute of Automation, CAS, Beijing, China 2 School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China 3 Beijing Academy of Artificial Intelligence, Beijing, 100084, China {yiming.ju, yzzhang, zhao.yang}@nlpr.ia.ac.cn {zhongtao.jiang, kliu, jzhao}@nlpr.ia.ac.cn Abstract Modern deep learning models are notoriously opaque, which has motivated the development of methods for interpreting how deep models predict. This goal is usually approached with at- tribution method, which assesses the influence of features on model predictions. As an expla- nation method, the evaluation criteria of attri- bution methods is how accurately it reflects the actual reasoning process of the model (faithful- ness). Meanwhile, since the reasoning process of deep models is inaccessible, researchers de- sign various evaluation methods to demonstrate their arguments. However, some crucial logic traps in these evaluation methods are ignored in most works, causing inaccurate evaluation and unfair comparison. This paper systematically reviews existing methods for evaluating attri- bution scores and summarizes the logic traps in these methods. We further conduct exper- iments to demonstrate the existence of each logic trap. Through both theoretical and ex- perimental analysis, we hope to increase atten- tion on the inaccurate evaluation of attribution scores. Moreover, with this paper, we suggest stopping focusing on improving performance under unreliable evaluation systems and start- ing efforts on reducing the impact of proposed logic traps. 1 Introduction The opaqueness of deep models has grown in tan- dem with their power (Doshi-Velez and Kim, 2017), which has motivated efforts to interpret how these black-box models work (Sundararajan et al., 2017; Belinkov and Glass, 2019). Post-hoc explana- tion aims to explain a trained model and reveal how the model arrives at a decision (Jacovi and Goldberg, 2020; Molnar, 2020). This goal is usu- ally approached with attribution method, which assesses the influence of features on model pre- dictions as shown in Figure 1. Recent years have witnessed an increasing number of attribution meth- ods being developed. For example, Erasure-based method calculate attribution scores by measuring the change of output after removing target features (Li et al., 2016; Feng et al., 2018; Chen et al., 2020); Gradient-based method uses gradients to study the influence of features on model predictions (Sun- dararajan et al., 2017; Wallace et al., 2019; Hao et al., 2020); Meanwhile, these methods also re- ceived much scrutiny, arguing that the generated attribution scores are fragile or unreliable (Alvarez- Melis and Jaakkola, 2018; Pruthi et al., 2020; Wang et al., 2020; Slack et al., 2020). Figure 1: An example of attribution explanations, which assesses the influence of each token on the predictions of a binary sentiment classification task. The saturation of the colors signifies the magnitude of the influence. As an explanation method, the evaluation crite- …

Content truncated to 500 words