Neuron-level Interpretation of Deep NLP Models: A Survey

Neuron-level Interpretation of Deep NLP Models: A Survey Hassan Sajjad♣∗ Nadir Durrani♠∗ Fahim Dalvi♠∗ ♣Faculty of Computer Science, Dalhousie University, Canada† ♠Qatar Computing Research Institute, HBKU, Doha, Qatar hsajjad@dal.ca, {ndurrani, faimaduddin}@hbku.edu.qa Abstract The proliferation of Deep Neural Networks in various domains has seen an increased need for interpretability of these models. Prelimi- nary work done along this line, and papers that surveyed such, are focused on high-level rep- resentation analysis. However, a recent branch of work has concentrated on interpretability at a more granular level of analyzing neurons within these models. In this paper, we survey the work done on neuron analysis including: i) methods to discover and understand neurons in a network; ii) evaluation methods; iii) major findings including cross architectural compar- isons that neuron analysis has unraveled; iv) applications of neuron probing such as: con- trolling the model, domain adaptation, and so forth; and v) a discussion on open issues and future research directions. 1 Introduction Models trained using Deep Neural Networks (DNNs) have constantly pushed the state-of- the-art in various Natural Language Processing (NLP) problems, for example, Language Model- ing (Mikolov et al., 2013; Devlin et al., 2019) and Machine Translation (Sutskever et al., 2014; Bahdanau et al., 2014) to name a few. Despite this remarkable revolution, the black-box nature of DNNs has remained a major bottleneck in their large scale adaptability—especially in the appli- cations where fairness, trust, accountability, relia- bility, and ethical decision-making are considered critically important metrics or at least as important as the model’s performance (Lipton, 2016). This opaqueness of DNNs has spurred a new area of research to analyze and understand these models. A plethora of papers have been written in the past five years on interpreting deep NLP ∗The authors contributed equally. †The work was done while the author was at QCRI. models and to answer one question in particular: What knowledge is learned within representa- tions? We term this work as the Representation Analysis. Representation Analysis thrives on post-hoc decomposability, where we analyze the embed- dings to uncover linguistic (and non-linguistic) concepts1 that are captured as the network is trained towards an NLP task (Adi et al., 2016; Belinkov et al., 2017a; Conneau et al., 2018; Liu et al., 2019; Tenney et al., 2019). A majority of the work on Representation Analysis has focused on a holistic view of the representations, namely, how much knowledge of a certain concept is learned within representations as a whole (See Belinkov et al. (2020a) for a survey done on this line of work). Recently, a more fine-grained neu- ron interpretation has started to gain attention. In addition to the holistic view of the represen- tation, Neuron Analysis provides insight into a fundamental question: How is knowledge struc- tured within these representations? In particular, it targets questions such as: • What concepts are learned within neurons of the network? • Are there neurons that specialize in learning particular concepts? • How localized/distributed and redundantly is the knowledge preserved within neurons of the network? Answers …

Content truncated to 500 words