Interpreting Neural Networks with Nearest Neighbors

Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 136–144 Brussels, Belgium, November 1, 2018. c⃝2018 Association for Computational Linguistics 136 Interpreting Neural Networks with Nearest Neighbors Eric Wallace∗, Shi Feng∗, Jordan Boyd-Graber University of Maryland {ewallac2,shifeng,jbg}@umiacs.umd.edu Abstract Local model interpretation methods explain individual predictions by assigning an impor- tance value to each input feature. This value is often determined by measuring the change in confidence when a feature is removed. How- ever, the confidence of neural networks is not a robust measure of model uncertainty. This is- sue makes reliably judging the importance of the input features difficult. We address this by changing the test-time behavior of neural net- works using Deep k-Nearest Neighbors. With- out harming text classification accuracy, this algorithm provides a more robust uncertainty metric which we use to generate feature im- portance values. The resulting interpretations better align with human perception than base- line methods. Finally, we use our interpreta- tion method to analyze model predictions on dataset annotation artifacts. 1 Introduction The growing use of neural networks in sensitive domains such as medicine, finance, and security raises concerns about human trust in these ma- chine learning systems. A central question is test- time interpretability: how can humans understand the reasoning behind model predictions? A common way to interpret neural network predictions is to identify the most important in- put features. For instance, a visual saliency map that highlights important pixels in an image (Sun- dararajan et al., 2017) or words in a sentence (Li et al., 2016). Given a model’s test prediction, the importance of each input feature is the change in model confidence when that feature is removed. However, neural network confidence is not a proper measure of model uncertainty (Guo et al., 2017). This issue is emphasized when models make highly confident predictions on inputs that ∗⋆Equal contribution are completely void of information, for example, images of pure noise (Goodfellow et al., 2015) or meaningless text snippets (Feng et al., 2018). Consequently, a model’s confidence may not prop- erly reflect whether discriminative input features are present. This issue makes it difficult to re- liably judge the importance of each input fea- ture using common confidence-based interpreta- tion methods (Feng et al., 2018). To address this, we apply Deep k-Nearest Neighbors (DKNN) (Papernot and McDaniel, 2018) to neural models for text classification. Concretely, predictions are no longer made with a softmax classifier, but using the labels of the train- ing examples whose representations are most sim- ilar to the test example (Section 3). This provides an alternative metric for model uncertainty, con- formity, which measures how much support a test prediction has by comparing its hidden represen- tations to the training data. This representation- based uncertainty measurement can be used in combination with existing interpretation methods, such as leave-one-out (Li et al., 2016), to better identify important input features. We combine DKNN with CNN and LSTM models on six NLP text classification tasks, includ- ing sentiment analysis …

Content truncated to 500 words