On the Lack of Robust Interpretability of Neural Text Classifiers

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3730–3740 August 1–6, 2021. ©2021 Association for Computational Linguistics 3730 On the Lack of Robust Interpretability of Neural Text Classifiers Muhammad Bilal Zafar Amazon zafamuh@amazon.com Michele Donini Amazon donini@amazon.com Dylan Slack∗ University of California, Irvine dslack@uci.edu C´edric Archambeau Amazon cedrica@amazon.com Sanjiv Das Amazon & Santa Clara University sanjivda@amazon.com Krishnaram Kenthapadi Amazon kenthk@amazon.com Abstract With the ever-increasing complexity of neu- ral language models, practitioners have turned to methods for understanding the predictions of these models. One of the most well- adopted approaches for model interpretability is feature-based interpretability, i.e., ranking the features in terms of their impact on model predictions. Several prior studies have focused on assessing the fidelity of feature-based in- terpretability methods, i.e., measuring the im- pact of dropping the top-ranked features on the model output. However, relatively little work has been conducted on quantifying the robust- ness of interpretations. In this work, we as- sess the robustness of interpretations of neural text classifiers, specifically, those based on pre- trained Transformer encoders, using two ran- domization tests. The first compares the in- terpretations of two models that are identical except for their initializations. The second measures whether the interpretations differ be- tween a model with trained parameters and a model with random parameters. Both tests show surprising deviations from expected be- havior, raising questions about the extent of in- sights that practitioners may draw from inter- pretations. 1 Introduction In recent years, large scale language models like BERT and RoBERTa have helped achieve new state-of-the-art performance on a variety of NLP tasks (Devlin et al., 2019; Liu et al., 2019). While relying on vast amounts of training data and model capacity has helped increase their accuracy, the rea- soning of these models is often hard to comprehend. To this end, several techniques have been proposed to interpret the model predictions. Perhaps the most widely-adopted class of inter- pretability approaches is that of feature-based in- ∗Work done during internship at Amazon. terpretability where the goal is to assign an impor- tance score to each of the input features. These scores are also called feature attributions. Sev- eral methods in this class (e.g., SHAP (Lundberg and Lee, 2017), Integrated Gradients (Sundararajan et al., 2017)) possess desirable theoretical proper- ties making them attractive candidates for inter- pretability. Benchmarking analyses often show that these methods possess high fidelity, i.e., removing fea- tures marked important by the interpretability method from the input indeed leads to significant change in the model output as expected (Atanasova et al., 2020; Lundberg and Lee, 2017). However, relatively few investigations have been carried out to understand the robustness of feature attributions. To explore the robustness, we conduct two tests based on randomization: Different Initializations Test: This test opera- tionalizes the implementation invariance property of Sundararajan et al. (2017). Given an input, it compares the feature attributions between two models that are identical in every aspect—that is, trained with same architecture, with same data, and same learning schedule—except for their …

Content truncated to 500 words