A Diagnostic Study of Explainability Techniques for Text Classification

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 3256–3274, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 3256 A Diagnostic Study of Explainability Techniques for Text Classification Pepa Atanasova Jakob Grue Simonsen Christina Lioma Isabelle Augenstein Department of Computer Science University of Copenhagen Denmark {pepa, simonsen, c.lioma, augenstein}@di.ku.dk Abstract Recent developments in machine learning have introduced models that approach human performance at the cost of increased architec- tural complexity. Efforts to make the ratio- nales behind the models’ predictions transpar- ent have inspired an abundance of new ex- plainability techniques. Provided with an al- ready trained model, they compute saliency scores for the words of an input instance. How- ever, there exists no definitive guide on (i) how to choose such a technique given a particular application task and model architecture, and (ii) the benefits and drawbacks of using each such technique. In this paper, we develop a comprehensive list of diagnostic properties for evaluating existing explainability techniques. We then employ the proposed list to compare a set of diverse explainability techniques on downstream text classification tasks and neu- ral network architectures. We also compare the saliency scores assigned by the explain- ability techniques with human annotations of salient input regions to find relations between a model’s performance and the agreement of its rationales with human ones. Overall, we find that the gradient-based explanations per- form best across tasks and model architectures, and we present further insights into the proper- ties of the reviewed explainability techniques. 1 Introduction Understanding the rationales behind models’ de- cisions is becoming a topic of pivotal importance, as both the architectural complexity of machine learning models and the number of their applica- tion domains increases. Having greater insight into the models’ reasons for making a particular predic- tion has already proven to be essential for discov- ering potential flaws or biases in medical diagno- sis (Caruana et al., 2015) and judicial sentencing (Rich, 2016). In addition, European law has man- Figure 1: Example of the saliency scores for the words (columns) of an instance from the Twitter Sentiment Extraction dataset. They are produced by the explain- ability techniques (rows) given a Transformer model. The first row is the human annotation of the salient words. The scores are normalized in the range [0, 1]. dated “the right . . . to obtain an explanation of the decision reached” (Regulation, 2016). Explainability methods attempt to reveal the rea- sons behind a model’s prediction for a single data point, as shown in Figure 1. They can be produced post-hoc, i.e., with already trained models. Such post-hoc explanation techniques can be applicable to one specific model (Martens et al., 2008; Wagner et al., 2019) or to a broader range thereof (Ribeiro et al., 2016; Lundberg and Lee, 2017). They can further be categorised as: employing model gra- dients (Sundararajan et al., 2017; Simonyan et al., 2013), being perturbation based (Shapley, 1953; Zeiler and Fergus, 2014) or providing explana- tions through model simplifications (Ribeiro et …

Content truncated to 500 words