Can We Really Trust Explanations? Evaluating the Stability of Feature Attribution Explanation Methods via Adversarial Attack

Can We Really Trust Explanations? Evaluating the Stability of Feature Attribution Explanation Methods via Adversarial Attack Zhao Yang1,2, Yuanzhe Zhang1,2, Zhongtao Jiang1,2, Yiming Ju1,2, Jun Zhao1,2, Kang Liu1,2,3∗ 1School of Artificial Intelligence, University of Chinese Academy of Sciences / Beijing, 100049, China 2National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences / Beijing, 100190, China 3Beijing Academy of Artificial Intelligence / Beijing, 100084, China {zhao.yang,yzzhang,zhongtao.jiang}@nlpr.ia.ac.cn {yiming.ju,jzhao,kliu}@nlpr.ia.ac.cn Abstract Explanations can increase the transparency of neural networks and make them more trustworthy. However, can we really trust explanations generated by the existing explanation methods? If the explanation methods are not stable enough, the credibility of the explanation will be greatly reduced. Previous studies seldom considered such an important issue. To this end, this paper proposes a new evaluation frame to evaluate the stability of current typical feature attribution explanation methods via textual adversarial attack. Our frame could generate adversarial exam- ples with similar textual semantics. Such adversarial examples will make the original models have the same outputs, but make most current explanation methods deduce completely different explanations. Under this frame, we test five classical explanation methods and show their perfor- mance on several stability-related metrics. Experimental results show our evaluation is effective and could reveal the stability performance of existing explanation methods. 1 Introduction Fueled by recent rapid development in deep learning, NLP systems have obtained promising results in several fields, such as medical, law and commerce (Rudin, 2019; Bommasani et al., 2021). However, besides the predicted results, users concern more on how these results are generated (Lipton, 2018). To this end, lots of emphases have been set upon the explanation methods for neural networks (Ribeiro et al., 2016; Li et al., 2016; Simonyan et al., 2013; Bastings et al., 2019). Although the current explanation methods have increased the transparency of the neural networks and provided explanations as supports for predicted results, most of them ignored important questions: are these methods reliable and the generated explanations really trustful? Besides the widely used focused properties of explanation methods, such as faithfulness, plausibility (Adebayo et al., 2018; Jacovi and Goldberg, 2020; Atanasova et al., 2020), readableness (Bastings et al., 2019) and compactness (Miller, 2019; Jiang et al., 2021), we believe stability is an important but often overlooked property (Robnik- ˇSikonja and Bohanec, 2018). When we put a small perturbation on the input, which would not change the input semantic and the output of the original model, we believe that the explanation method is not stable enough when we obtain the same outputs with quite different explanations. For example, Figure 1 shows all results of major explanation methods would change when we just replace fine by refined, including LIME (Ribeiro et al., 2016), Leave-one-out (Li et al., 2016), Vanilla Gradient (Simonyan et al., 2013), Smooth Gradient (Smilkov et al., 2017), Integrated Gradient (Sundararajan et al., 2017). To fulfill the stability testing, we intuitively consider existing word-substitution based textual adver- sarial attack methods0(Ren et al., 2019; Zang et al., 2020), since it is …

Content truncated to 500 words