The Solvability of Interpretability Evaluation Metrics

Findings of the Association for Computational Linguistics: EACL 2023, pages 2399–2415 May 2-6, 2023 ©2023 Association for Computational Linguistics The Solvability of Interpretability Evaluation Metrics Yilun Zhou Julie Shah MIT CSAIL {yilun,julie_a_shah}@csail.mit.edu https://yilunzhou.github.io/solvability/ Abstract Feature attribution methods are popular for ex- plaining neural network predictions, and they are often evaluated on metrics such as compre- hensiveness and sufficiency. In this paper, we highlight an intriguing property of these met- rics: their solvability. Concretely, we can de- fine the problem of optimizing an explanation for a metric, which can be solved by beam search. This observation leads to the obvi- ous yet unaddressed question: why do we use explainers (e.g., LIME) not based on solving the target metric, if the metric value repre- sents explanation quality? We present a series of investigations showing strong performance of this beam search explainer and discuss its broader implication: a definition-evaluation duality of interpretability concepts. We im- plement the explainer and release the Python solvex package for models of text, image and tabular domains. 1 Introduction For neural network models deployed in high stakes domains, the explanations for predictions are often as important as the predictions themselves. For example, a skin cancer detection model may work by detecting surgery markers (Winkler et al., 2019) and an explanation that reveals this spurious correlation is highly valuable. However, evaluating the correctness (or faithfulness) of explanations is fundamentally ill-posed: because the explanations are used to help people understand the reasoning of the model, we cannot check it against the ground truth reasoning, as the latter is not available. As a result, correctness evaluations typically em- ploy certain alternative metrics. For feature attribu- tion explanations, they work under a shared prin- ciple: changing an important feature should have a large impact on the model prediction. Thus, the quality of the explanation is defined by different for- mulations of the model prediction change, resulting in various metrics such as comprehensiveness and Identify what feature importance means Propose the explainer to reflect the meaning Evaluate the explainer on the quality metric Better metric value ⟺ better explainer Start Explicitly search for the explanation that optimizes the target quality metric Optimal metric value by construction Heuristic Search-Based Figure 1: Left: the current process of developing new explainers. Right: the natural implication following our observation that evaluation metrics are solvable. sufficiency (DeYoung et al., 2020). To develop new explanation methods (Fig. 1, left), people gener- ally identify a specific notion of feature importance (e.g., local sensitivity), propose the correspond- ing explainer (e.g., gradient saliency (Simonyan et al., 2013)), evaluate it on one or more metrics, and claim its superiority based on favorable results vs. baseline explainers. We call these explainers heuristic as they are motivated by pre-defined no- tions of feature importance. In this paper, we show that all these metrics are solvable, in that we can define an explanation as the one that optimizes a metric value and search for it. The obvious question is then: if we take a specific target …

Content truncated to 500 words