Finding Skill Neurons in Pre-trained Transformer-based Language Models

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11132 - 11152 December 7-11, 2022 ©2022 Association for Computational Linguistics Finding Skill Neurons in Pre-trained Transformer-based Language Models Xiaozhi Wang1∗, Kaiyue Wen2∗, Zhengyan Zhang1, Lei Hou1,3†, Zhiyuan Liu1,3†, Juanzi Li1,3 1Department of Computer Science and Technology, BNRist; 2Institute for Interdisciplinary Information Sciences; 3KIRC, Institute for Artificial Intelligence, Tsinghua University, Beijing, 100084, China {wangxz20,wenky20}@mails.tsinghua.edu.cn Abstract Transformer-based pre-trained language mod- els have demonstrated superior performance on various natural language processing tasks. However, it remains unclear how the skills re- quired to handle these tasks distribute among model parameters. In this paper, we find that after prompt tuning for specific tasks, the activa- tions of some neurons within pre-trained Trans- formers1 are highly predictive of the task labels. We dub these neurons skill neurons and confirm they encode task-specific skills by finding that: (1) Skill neurons are crucial for handling tasks. Performances of pre-trained Transformers on a task significantly drop when corresponding skill neurons are perturbed. (2) Skill neurons are task-specific. Similar tasks tend to have sim- ilar distributions of skill neurons. Furthermore, we demonstrate the skill neurons are most likely generated in pre-training rather than fine- tuning by showing that the skill neurons found with prompt tuning are also crucial for other fine-tuning methods freezing neuron weights, such as the adapter-based tuning and BitFit. We also explore the applications of skill neurons, including accelerating Transformers with net- work pruning and building better transferability indicators. These findings may promote fur- ther research on understanding Transformers. The source code can be obtained from https: //github.com/THU-KEG/Skill-Neuron. 1 Introduction Pre-trained language models (PLMs), mostly based on Transformer architecture (Vaswani et al., 2017), have achieved remarkable performance on broad and diverse natural language processing (NLP) tasks (Han et al., 2021). However, it remains un- clear how the skills required to handle these tasks distribute among model parameters. Are there ∗ indicates equal contribution. † Corresponding author: Z.Liu and L.Hou. 1For brevity, Transformer-based language models are often referred to as Transformers in this paper. 0.16 0.14 0.12 0.10 0.08 0.06 0.04 Activation 100 101 102 #Sentences Positive Negative Figure 1: Histogram of activation of a neuron within RoBERTaBASE on positive-label (blue) and negative- label (orange) sentences in SST-2 validation set. specific neurons within pre-trained Transformers encoding these skills? Progress on this problem may help to understand the working mechanisms of pre-trained Transformers (Zeiler and Fergus, 2014; Karpathy et al., 2015; Bau et al., 2020; Suau et al., 2020), intervene model behaviors (Bau et al., 2018; Mitchell et al., 2021), and improve model efficiency (Dalvi et al., 2020; Zhang et al., 2021). Prompt tuning (Li and Liang, 2021; Lester et al., 2021) prepends some trainable embeddings, i.e., soft prompts, into the inputs and adapts PLMs to handle tasks by only tuning the soft prompts while freezing all the PLM parameters. It has attracted wide attention recently as a promising parameter- efficient fine-tuning methods (Su et al., 2021; Liu et al., 2022). In this paper, we …

Content truncated to 500 words