Interpretation of NLP models through input marginalization

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 3154–3167, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 3154 Interpretation of NLP models through input marginalization Siwon Kim Jihun Yi Eunji Kim Sungroh Yoon∗ Data Science and Artificial Intelligence Laboratory ECE, Interdisciplinary Program in AI, and Institute of Engineering Research Seoul National University, Seoul 08826, South Korea {tuslkkk, t080205, kce407, sryoon}@snu.ac.kr Abstract To demystify the “black box” property of deep neural networks for natural language process- ing (NLP), several methods have been pro- posed to interpret their predictions by measur- ing the change in prediction probability after erasing each token of an input. Since existing methods replace each token with a predefined value (i.e., zero), the resulting sentence lies out of the training data distribution, yielding mis- leading interpretations. In this study, we raise the out-of-distribution problem induced by the existing interpretation methods and present a remedy; we propose to marginalize each token out. We interpret various NLP models trained for sentiment analysis and natural language in- ference using the proposed method. 1 Introduction The advent of deep learning has greatly improved the performances of natural language processing (NLP) models. Consequently, the models are be- coming more complex (Yang et al., 2019; Liu et al., 2019), rendering it difficult to understand the ratio- nale behind their predictions. To use deep neural networks (DNNs) for making high-stakes decisions, the interpretability must be guaranteed to instill the trust in the public. Hence, various attempts have been undertaken to provide an interpretation along with a prediction (Gilpin et al., 2018). Research in computer vision aims to interpret a target model by measuring attribution scores, i.e., how much each pixel in an input image contributes to the final prediction (Simonyan et al., 2013; Arras et al., 2017; Zeiler and Fergus, 2014; Lundberg and Lee, 2017). Since a pixel of an image corresponds to a token in a sentence, the attribution score of each token can provide an insight into the NLP model’s internal reasoning process. A straightfor- ward approach is to ask, “How would the model reaction change if each token was not there?” and *Correspondence to: Sungroh Yoon (sryoon@snu.ac.kr) (a) Original sentence (b) Existing erasure scheme (c) Input marginalization (Ours) Prediction 0.979 _ _ It’s also, clearly, great fun. It’s also, clearly [PAD] great fun. Likelihood 0.890 0.978 It’s also, clearly great fun. , It’s also, clearly great fun. a It’s also, clearly great fun. not ... It’s also, clearly great fun. 0.979 0.977 0.457 ... 0.009 0.002 ... Figure 1: Given the original sentence (a), the existing erasure scheme (b) replaces each token with zero, i.e., [PAD] token. Our method (c) marginalizes each token out considering the likelihoods of candidate tokens. the change can be measured by the difference in softmax probabilities after erasing each token. Li et al. (2016) proposed to erase each token by re- placing it with a predefined value, i.e., zero. This became a representative method for interpreting NLP models, followed by several papers …

Content truncated to 500 words