What Part of the Neural Network Does This? Understanding LSTMs by Measuring and Dissecting Neurons

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 5823–5830, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 5823 What Part of the Neural Network Does This? Understanding LSTMs by Measuring and Dissecting Neurons Ji Xin, Jimmy Lin, and Yaoliang Yu David R. Cheriton School of Computer Science University of Waterloo {ji.xin,jimmylin,yaoliang.yu}@uwaterloo.ca Abstract Memory neurons of long short-term mem- ory (LSTM) networks encode and process in- formation in powerful yet mysterious ways. While there has been work to analyze their be- havior in carrying low-level information such as linguistic properties, how they directly con- tribute to label prediction remains unclear. We find inspiration from biologists and study the affinity between individual neurons and labels, propose a novel metric to quantify the sensi- tivity of neurons to each label, and conduct experiments to show the validity of our pro- posed metric. We discover that some neu- rons are trained to specialize on a subset of labels, and while dropping an arbitrary neu- ron has little effect on the overall accuracy of the model, dropping label-specialized neurons predictably and significantly degrades predic- tion accuracy on the associated label. We fur- ther examine the consistency of neuron-label affinity across different models. These obser- vations provide insight into the inner mecha- nisms of LSTMs. 1 Introduction In recent years, the application of deep learning to natural language processing (NLP) has been a success. Many consider the employment of distributed representations to be one of the rea- sons for deep learning’s success (LeCun et al., 2015; Young et al., 2018). However, how these distributed representations encode information in deep neural networks, especially long short-term memory (LSTM) networks that are prevalent in NLP, still remains unclear (Feng et al., 2018). One of the potential ways to understand how neural networks function is to analyze the behavior of individual neurons that carry the distributed rep- resentation. While there have been a number of works that analyze low-level information stored in individual LSTM neurons, such as linguistic prop- erties (Bau et al., 2019; Qian et al., 2016), syntax of source code (Karpathy et al., 2015), and senti- ment (Radford et al., 2017), how each neuron con- tributes directly to the final classification layer re- mains unclear. We find inspiration to analyze individual neu- rons of LSTMs from how biologists analyze neu- rons of roundworms (White et al., 1986). Biolog- ical neural systems consist of a huge number of neurons, and can react to the environment in com- plicated ways. Biologists start with analyzing ba- sic components of reactions, what stimuli trigger them, and which neurons are excited during the process. To verify the relationship between stim- uli, neurons, and reactions, biologists further dis- sect neurons which are correlated with specific ba- sic reactions, and see if the reaction still occurs for the same stimuli. We adopt the same methodology to study LSTMs, using a representative task in NLP: named-entity recognition (NER) …

Content truncated to 500 words