Deep Neural Model Inspection and Comparison via Functional Neuron Pathways

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5754–5764 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 5754 Deep Neural Model Inspection and Comparison via Functional Neuron Pathways James Fiacco Language Technologies Institute Carnegie Mellon University jfiacco@cs.cmu.edu Samridhi Choudhary∗ Alexa Machine Learning Amazon samridhc@amazon.com Carolyn P. Ros´e Language Technologies Institute Carnegie Mellon University cprose@cs.cmu.edu Abstract We introduce a general method for the interpre- tation and comparison of neural models. The method is used to factor a complex neural model into its functional components, which are comprised of sets of co-firing neurons that cut across layers of the network architecture, and which we call neural pathways. The func- tion of these pathways can be understood by identifying correlated task level and linguistic heuristics in such a way that this knowledge acts as a lens for approximating what the net- work has learned to apply to its intended task. As a case study for investigating the utility of these pathways, we present an examination of pathways identified in models trained for two standard tasks, namely Named Entity Recogni- tion and Recognizing Textual Entailment. 1 Introduction Interpretation of neural models is a difficult task because the knowledge learned within neural net- works is distributed across hundreds of thousands of parameters. Interpreting the significance of any individual neuron is tantamount to reconstructing a forest based on a single pine needle. More specifi- cally, the contribution of each individual neuron is a minuscule part in the overall representation of the learned solution, and the mapping between neurons and function may be many-to-many (Goodfellow et al., 2016). As a response to this, the contri- bution of this paper is a new method of network interpretation that enables a more abstract view of what a network has learned, which we refer to as neural pathways. In this approach, inspired by the concept of biological neural pathways used in neuroscience research to understand physical brain function (Kennedy et al., 1975), a network is fac- tored into functional groups of co-firing neurons ∗ Work was done as a graduate student at Carnegie Mellon University. that cut across layers in a complex network archi- tecture. Rather than attempt interpretation of the activation pattern through a single neuron at a time, we instead attempt interpretation of a functional group of neurons where the activation pattern of the group can then be more effectively associated with task and linguistic knowledge. This enables understanding the neuron groups as working to- gether to accomplish a comprehensible sub-task. These pathways help conceptualize what task and linguistic knowledge a model may be using in an approximate way, the benefit of which is that it does not depend on an isomorphism between network architectures. This method, which can be applied simply in a purely post-hoc analysis, independent of the train- ing process, can enable both understanding of in- dividual models and comparison across models. The interpretation process enables investigation of which identified functional groups correspond to linguistic …

Content truncated to 500 words