A Comparative Study of Faithfulness Metrics for Model Interpretability Methods

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 5029 - 5038 May 22-27, 2022 c⃝2022 Association for Computational Linguistics A Comparative Study of Faithfulness Metrics for Model Interpretability Methods Chun Sik Chan, Huanqi Kong, Guanqing Liang Wisers AI Lab, Wisers Information Limited {tonychan, katekong, quincyliang}@wisers.com Abstract Interpretation methods to reveal the internal reasoning processes behind machine learning models have attracted increasing attention in recent years. To quantify the extent to which the identified interpretations truly reflect the intrinsic decision-making mechanisms, various faithfulness evaluation metrics have been pro- posed. However, we find that different faithful- ness metrics show conflicting preferences when comparing different interpretations. Motivated by this observation, we aim to conduct a com- prehensive and comparative study of the widely adopted faithfulness metrics. In particular, we introduce two assessment dimensions, namely diagnosticity and time complexity. Diagnostic- ity refers to the degree to which the faithfulness metric favours relatively faithful interpretations over randomly generated ones, and time com- plexity is measured by the average number of model forward passes. According to the ex- perimental results, we find that sufficiency and comprehensiveness metrics have higher diag- nosticity and lower time complexity than the other faithfulness metrics. 1 Introduction NLP has made tremendous progress in recent years. However, the increasing complexity of the models makes their behaviour difficult to interpret. To disclose the rationale behind the models, various interpretation methods have been proposed. Interpretation methods can be broadly classified into two categories: model-based methods and post- hoc methods. Model-based approaches refer to designing simple and white-box machine learning models whose internal decision logic can be easily interpreted, such as linear regression models, deci- sion trees, etc. A post-hoc method is applied after model training and aims to disclose the relation- ship between feature values and predictions. As pretrained language models (Devlin et al., 2019a; Liu et al., 2019; Brown et al., 2020) become more popular, deep learning models are becoming more and more complex. Therefore, post-hoc methods are the only option for model interpretations. Post- hoc interpretation methods can be divided into two categories: gradient-based (Simonyan et al., 2014; Sundararajan et al., 2017; Shrikumar et al., 2019) and perturbation-based (Robnik-Šikonja and Kononenko, 2008; Zeiler and Fergus, 2013; Ribeiro et al., 2016). Gradient-based methods assume the model is differentiable and attempt to interpret the model outputs through the gradient information. Perturbation-based methods interpret model out- puts by perturbing the input data. To verify whether, and to what extent, the inter- pretations reflect the intrinsic reasoning process, various faithfulness metrics have been proposed. Most faithfulness metrics use a removal-based cri- terion, i.e., removing or retaining only the impor- tant tokens identified by the interpretation and ob- serving the changes in model outputs (Serrano and Smith, 2019; Chrysostomou and Aletras, 2021; Ar- ras et al., 2017; DeYoung et al., 2020). However, we observe that the existing faithful- ness metrics are not always consistent with each other and even lead to contradictory conclusions. As shown in the example …

Content truncated to 500 words