Analyzing Individual Neurons in Pre-trained Language Models

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 4865–4880, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 4865 Analyzing Individual Neurons in Pre-trained Language Models Nadir Durrani Hassan Sajjad Fahim Dalvi Yonatan Belinkov* {ndurrani,hsajjad,faimaduddin}@hbku.edu.qa Qatar Computing Research Institute, HBKU Research Complex, Doha 5825, Qatar *MIT Computer Science and Artificial Intelligence Laboratory and Harvard John A. Paulson School of Engineering and Applied Sciences, Cambridge, MA, USA belinkov@csail.mit.edu Abstract While a lot of analysis has been carried to demonstrate linguistic knowledge captured by the representations learned within deep NLP models, very little attention has been paid towards individual neurons. We carry out a neuron-level analysis using core linguistic tasks of predicting morphology, syntax and se- mantics, on pre-trained language models, with questions like: i) do individual neurons in pre- trained models capture linguistic information? ii) which parts of the network learn more about certain linguistic phenomena? iii) how dis- tributed or focused is the information? and iv) how do various architectures differ in learning these properties? We found small subsets of neurons to predict linguistic tasks, with lower level tasks (such as morphology) localized in fewer neurons, compared to higher level task of predicting syntax. Our study reveals inter- esting cross architectural comparisons. For ex- ample, we found neurons in XLNet to be more localized and disjoint when predicting proper- ties compared to BERT and others, where they are more distributed and coupled. 1 Introduction Transformer-based neural language models have constantly pushed the state-of-the-art in down- stream NLP tasks such as Question Answering, Textual Entailment, etc. (Rajpurkar et al., 2016; Wang et al., 2018). Central to this revolution is the contextualized embedding, where each word is as- signed a vector based on the entire input sequence, allowing it to capture not only a static semantic meaning but also a contextualized meaning. Previous work on analyzing neural networks showed that while learning rich NLP tasks such as machine translation and language modeling, these deep models capture fundamental linguistic phe- nomena such as word morphology, syntax and vari- ous other relevant properties of interest (Shi et al., 2016; Adi et al., 2016; Belinkov et al., 2017a,b; Dalvi et al., 2017; Blevins et al., 2018). More recently Liu et al. (2019) and Tenney et al. (2019) used probing classifiers to analyze pre- trained neural language models on a variety of se- quence labeling tasks and demonstrated that contex- tualized representations encode useful, transferable features of language. While most of the previous studies emphasize and analyze representations as a whole, very little work has been carried to analyze individual neurons in deep NLP models. Studying individual neurons can facilitate under- standing of the inner workings of neural networks (Karpathy et al., 2015; Dalvi et al., 2019; Suau et al., 2020) and have other potential benefits such as controlling bias and manipulating system’s be- haviour (Bau et al., 2019), model distillation and compression (Rethmeier et al., 2020), efficient fea- ture selection (Dalvi et al., 2020), and guiding ar- chitectural search. In this …

Content truncated to 500 words