Evaluating Saliency Methods for Neural Language Models

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5034–5052 June 6–11, 2021. ©2021 Association for Computational Linguistics 5034 Evaluating Saliency Methods for Neural Language Models Shuoyang Ding Philipp Koehn Center for Language and Speech Processing Johns Hopkins University {dings, phi}@jhu.edu Abstract Saliency methods are widely used to interpret neural network predictions, but different vari- ants of saliency methods often disagree even on the interpretations of the same prediction made by the same model. In these cases, how do we identify when are these interpretations trustworthy enough to be used in analyses? To address this question, we conduct a compre- hensive and quantitative evaluation of saliency methods on a fundamental category of NLP models: neural language models. We evaluate the quality of prediction interpretations from two perspectives that each represents a desir- able property of these interpretations: plausi- bility and faithfulness. Our evaluation is con- ducted on four different datasets constructed from the existing human annotation of syntac- tic and semantic agreements, on both sentence- level and document-level. Through our evalua- tion, we identified various ways saliency meth- ods could yield interpretations of low qual- ity. We recommend that future work deploy- ing such methods to neural language models should carefully validate their interpretations before drawing insights. 1 Introduction While neural network models for Natural Language Processing (NLP) have recently become popular, a general complaint is that their internal decision mechanisms are hard to understand. To alleviate this problem, recent work has deployed interpreta- tion methods on top of the neural network models. Among them, there is a category of interpretation methods called saliency method that is especially widely adopted (Li et al., 2016a,b; Arras et al., 2016, 2017; Mudrakarta et al., 2018; Ding et al., 2019). At a very high level, these methods assign an importance score to each feature in the input fea- ture set F, regarding a specific prediction y made by a neural network model M. Such feature impor- tance scores can hopefully shed light on the neural network models’ internal decision mechanism. V U.S. companies wanting to expand in Europe SG U.S. companies wanting to expand in Europe IG U.S. companies wanting to expand in Europe Table 1: An example from our evaluation where dif- ferent saliency methods assign different importance scores for the same model (Transformer language model) and the same next word prediction (are). V, SG and IG are different saliency methods (see Section 2). The tints of green and yellow mark the magnitude of positive and negative importance scores, respectively. While analyzing saliency interpretations uncov- ers useful insights for their respective task of in- terest, different saliency methods often give differ- ent interpretations even when the internal decision mechanism remains the same (with F, y and M held constant), as exemplified in Table 1. Even so, most existing work that deploys these meth- ods often makes an ungrounded assumption that a specific saliency method can reliably uncover the internal model decision …

Content truncated to 500 words