On the Sensitivity and Stability of Model Interpretations in NLP

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 2631 - 2647 May 22-27, 2022 c⃝2022 Association for Computational Linguistics On the Sensitivity and Stability of Model Interpretations in NLP Fan Yin, Zhouxing Shi, Cho-Jui Hsieh, and Kai-Wei Chang University of California, Los Angeles {fanyin20, zshi, chohsieh, kwchang}@cs.ucla.edu; Abstract Recent years have witnessed the emergence of a variety of post-hoc interpretations that aim to uncover how natural language processing (NLP) models make predictions. Despite the surge of new interpretation methods, it remains an open problem how to define and quantita- tively measure the faithfulness of interpreta- tions, i.e., to what extent interpretations reflect the reasoning process by a model. We pro- pose two new criteria, sensitivity and stability, that provide complementary notions of faith- fulness to the existed removal-based criteria. Our results show that the conclusion for how faithful interpretations are could vary substan- tially based on different notions. Motivated by the desiderata of sensitivity and stability, we introduce a new class of interpretation meth- ods that adopt techniques from adversarial ro- bustness. Empirical results show that our pro- posed methods are effective under the new crite- ria and overcome limitations of gradient-based methods on removal-based criteria. Besides text classification, we also apply interpretation methods and metrics to dependency parsing. Our results shed light on understanding the di- verse set of interpretations. 1 Introduction As complex NLP models are widely deployed in real-world applications, there is an increasing in- terest in understanding how these models come to certain decisions. As a result, the line of research on interpretation techniques grows rapidly, facilitat- ing a broad range of model analysis, from building user trust on models (Ribeiro et al., 2016; Hase and Bansal, 2020) to exposing subtle biases (Zhao et al., 2017; Doshi-Velez and Kim, 2017). In this paper, we focus on post-hoc interpreta- tions in NLP. Given a trained model and a specific input text, post-hoc interpretations assign an im- portance score to each token in the input which indicates its contribution to the model output. Cur- rent methods in this direction can be roughly di- vided into three categories: gradient-based methods (Simonyan et al., 2014; Li et al., 2016); reference- based methods (Sundararajan et al., 2017; Shriku- mar et al., 2017); and perturbation-based methods (Zeiler and Fergus, 2014; Ribeiro et al., 2016). Despite the emergence of new techniques, one critical issue is that there is little consensus on how to define and evaluate the faithfulness of these techniques, i.e., whether they reflect the true rea- soning process by a model. A widely employed criterion, especially in NLP, is the removal-based criterion (DeYoung et al., 2020), which removes or only preserves a set of tokens given by interpreta- tions and measures how much the model prediction would change. However, as pointed out in prior work (Bastings and Filippova, 2020; Ancona et al., 2018), the corrupted version of an input produced during evaluations falls out of the distribution that models are trained on, …

Content truncated to 500 words