Knowledge Neurons in Pretrained Transformers

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 8493 - 8502 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Knowledge Neurons in Pretrained Transformers Damai Dai†‡∗, Li Dong‡, Yaru Hao‡, Zhifang Sui†, Baobao Chang†, Furu Wei‡ †MOE Key Lab of Computational Linguistics, Peking University ‡Microsoft Research {daidamai,szf,chbb}@pku.edu.cn {lidong1,yaruhao,fuwei}@microsoft.com Abstract Large-scale pretrained language models are surprisingly good at recalling factual knowl- edge presented in the training corpus (Petroni et al., 2019; Jiang et al., 2020b). In this pa- per, we present preliminary studies on how fac- tual knowledge is stored in pretrained Trans- formers by introducing the concept of knowl- edge neurons. Specifically, we examine the fill-in-the-blank cloze task for BERT. Given a relational fact, we propose a knowledge at- tribution method to identify the neurons that express the fact. We find that the activation of such knowledge neurons is positively cor- related to the expression of their correspond- ing facts. In our case studies, we attempt to leverage knowledge neurons to edit (such as update, and erase) specific factual knowledge without fine-tuning. Our results shed light on understanding the storage of knowledge within pretrained Transformers. The code is available at https://github.com/ Hunter-DDM/knowledge-neurons. 1 Introduction Large-scale pretrained Transformers (Devlin et al., 2019; Liu et al., 2019; Dong et al., 2019; Clark et al., 2020; Bao et al., 2020) are usually learned with a language modeling objective on large-scale corpora, such as Wikipedia, where exists oceans of factual knowledge. Pretrained language models naturally play as a free-text knowledge base by pre- dicting texts (Bosselut et al., 2019). Petroni et al. (2019) and Jiang et al. (2020b) probe factual knowl- edge stored in pretrained language models by fill- in-the-blank cloze queries. The evaluation shows that pretrained Transformers have a strong ability to recall factual knowledge without any fine-tuning. Roberts et al. (2020) use closed-book question an- swering to show that the larger a model is, the more knowledge it can store. However, most previous work focuses on evaluating the overall accuracy of ∗Contribution during internship at Microsoft Research. Self-Attention Layer Feed-Forward Network Knowledge Attribution Knowledge Neurons … … Q27 Ireland Q1761 Dublin capital P36 Factual Knowledge Hidden State Figure 1: Through knowledge attribution, we identify knowledge neurons that express a relational fact. text-form knowledge prediction. In this paper, we attempt to look deeper into pretrained Transformers and investigate how factual knowledge is stored. As shown in Figure 1, we propose a knowl- edge attribution method to identify the neurons that express a relational fact, where such neurons are named knowledge neurons. Specifically, we view feed-forward network (i.e., two-layer percep- tron) modules in Transformer as key-value memo- ries (Geva et al., 2020). For the example in Figure 1, the hidden state is fed into the first linear layer and activates knowledge neurons; then, the second linear layer integrates the corresponding memory vectors. The key-value-memory nature (Geva et al., 2020) inspires us to propose the knowledge attribu- tion method, which identifies knowledge neurons in feed-forward networks …

Content truncated to 500 words