Locally Aggregated Feature Attribution on Natural Language Model Understanding

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2189 - 2201 July 10-15, 2022 ©2022 Association for Computational Linguistics Locally Aggregated Feature Attribution on Natural Language Model Understanding Sheng Zhang1 Jin Wang2 Haitao Jiang3 Rui Song2,3 1AWS AI Labs, 2Amazon Core AI 3Department of Statistics, North Carolina State University {zshe, jiwngn}@amazon.com, {hjiang24, rsong}@ncsu.edu Abstract With the growing popularity of deep-learning models, model understanding becomes more important. Much effort has been devoted to demystify deep neural networks for better in- terpretability. Some feature attribution meth- ods have shown promising results in computer vision, especially the gradient-based methods where effectively smoothing the gradients with reference data is key to a robust and faithful result. However, direct application of these gradient-based methods to NLP tasks is not trivial due to the fact that the input consists of discrete tokens and the “reference” tokens are not explicitly defined. In this work, we propose Locally Aggregated Feature Attribu- tion (LAFA), a novel gradient-based feature attribution method for NLP models. Instead of relying on obscure reference tokens, it smooths gradients by aggregating similar reference texts derived from language model embeddings. For evaluation purpose, we also design experi- ments on different NLP tasks including Entity Recognition and Sentiment Analysis on public datasets as well as key feature detection on a constructed Amazon catalogue dataset. The su- perior performance of the proposed method is demonstrated through experiments. 1 Introduction With the growing popularity of deep-learning mod- els, model understanding becomes more and more critical in many folds. In one aspect, model un- derstanding helps us understand what the model is doing by identifying crucial features among un- structured raw data. For example, Shrikumar et al., 2017 utilized the model explainability technique to discover motifs in regulatory DNA elements from distinct molecular signatures in the field of Ge- nomics. In another aspect, model understanding helps people audit or debug the deep models. An interesting example is that Ribeiro et al. (Ribeiro et al., 2016) found that their image classification model sometimes misclassifies a husky as a wolf. The model explainability tool reveals that their model relies on the snow in the background rather than the appearance when distinguishing the two animals. More importantly, model understanding helps gain trust when making important decisions based on the model. In the NLP domain, deep language models are quickly evolving and show superior performance in various benchmark tasks. However, even experts struggle to understand the mechanism of complex language models. Much effort has been devoted to demystifying the “black box” of deep models. A natural idea is through feature attribution, explaining the model by attributing the prediction to each input feature according to how much it affects the model output, of which two main directions emerge. One is model agnostic approaches including Shapley regression values (Shapley, 1953) and LIME (Ribeiro et al., 2016). We can apply these methods regardless of the model structure, however, they could suffer from computational inefficiency in …

Content truncated to 500 words