ExSum: From Local Explanations to Model Understanding

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5359 - 5378 July 10-15, 2022 ©2022 Association for Computational Linguistics EXSUM: From Local Explanations to Model Understanding Yilun Zhou MIT CSAIL Marco Tulio Ribeiro Microsoft Research {yilun,julie_a_shah}@csail.mit.edu marcotcr@microsoft.com https://yilunzhou.github.io/exsum/ Julie Shah MIT CSAIL Abstract Interpretability methods are developed to un- derstand the working mechanisms of black- box models, which is crucial to their respon- sible deployment. Fulfilling this goal requires both that the explanations generated by these methods are correct and that people can easily and reliably understand them. While the for- mer has been addressed in prior work, the lat- ter is often overlooked, resulting in informal model understanding derived from a handful of local explanations. In this paper, we intro- duce explanation summary (EXSUM), a math- ematical framework for quantifying model un- derstanding, and propose metrics for its quality assessment. On two domains, EXSUM high- lights various limitations in the current prac- tice, helps develop accurate model understand- ing, and reveals easily overlooked properties of the model. We also connect understandabil- ity to other properties of explanations such as human alignment, robustness, and counterfac- tual similarity and plausibility. 1 Introduction Understanding a model’s behavior is often a prereq- uisite for deploying it in the real world, especially in high-stake scenarios such as financial, legal, and medical domains. Unfortunately, most high- performing models, such as neural networks, are black-boxes. Thus, model-agnostic interpretability techniques have been developed, with the majority being “local” – algorithms that produce an expla- nation for a specific input at a time (e.g., Li et al., 2016; Ribeiro et al., 2016). Even with these local explanations, there are still two hurdles to overcome before achieving the ulti- mate goal of complete understanding of a model. First, some local explanations may not correctly (or faithfully) represent the model’s reasoning process (Jacovi and Goldberg, 2020), as has been demon- strated both theoretically (Nie et al., 2018) and em- pirically (Adebayo et al., 2018) in prior work. As a Input Local Explanation Model Understanding The model thinks that adjectives are important. Model Correctness Understandability Figure 1: Local model explanations need to be both correct and easily understandable. While much prior work (e.g., Zhou et al., 2022) has studied the former property, this paper focuses on the latter, which has thus far been largely ignored. result, correctness evaluation has received much at- tention in the community (e.g., Samek et al., 2016; Arras et al., 2019; Zhou et al., 2022). Another mostly overlooked property of expla- nations is their understandability. As the model understanding pipeline depicted in Fig. 1 shows, explanations need to be both correct and easily un- derstandable, since even correct explanations are not as valuable if they lead to incorrect understand- ing. However, the concept of understandability has yet to be formalized, and instead users often de- rive model understanding from few examples in a non-rigorous (and potentially incorrect) manner. Consider the sentiment classification task shown in …

Content truncated to 500 words