Plug-and-Play Knowledge Injection for Pre-trained Language Models

Plug-and-Play Knowledge Injection for Pre-trained Language Models Zhengyan Zhang1∗, Zhiyuan Zeng1∗, Yankai Lin2,3, Huadong Wang1, Deming Ye1 Chaojun Xiao1, Xu Han1†, Zhiyuan Liu1,4,5†, Peng Li6, Maosong Sun1,4, Jie Zhou7 1NLP Group, DCST, IAI, BNRIST, Tsinghua University, Beijing 2Gaoling School of Artificial Intelligence, Renmin University of China, Beijing 3 Beijing Key Laboratory of Big Data Management and Analysis Methods 4International Innovation Center of Tsinghua University, Shanghai 5 Quan Cheng Laboratory 6 Institute for AI Industry Research (AIR), Tsinghua University, China 7 Pattern Recognition Center, WeChat AI, Tencent Inc {zy-z19,zengzy20}@mails.tsinghua.edu.cn {liuzy,sms}@tsinghua.edu.cn Abstract Injecting external knowledge can improve the performance of pre-trained language models (PLMs) on various downstream NLP tasks. However, massive retraining is required to deploy new knowledge injection methods or knowledge bases for downstream tasks. In this work, we are the first to study how to improve the flexibility and efficiency of knowledge injection by reusing existing down- stream models. To this end, we explore a new paradigm plug-and-play knowledge in- jection, where knowledge bases are injected into frozen existing downstream models by a knowledge plugin. Correspondingly, we pro- pose a plug-and-play injection method map- tuning, which trains a mapping of knowl- edge embeddings to enrich model inputs with mapped embeddings while keeping model pa- rameters frozen. Experimental results on three knowledge-driven NLP tasks show that existing injection methods are not suitable for the new paradigm, while map-tuning ef- fectively improves the performance of down- stream models. Moreover, we show that a frozen downstream model can be well adapted to different domains with different mapping networks of domain knowledge. Our code and models are available at https://github. com/THUNLP/Knowledge-Plugin. 1 Introduction Recent years have witnessed rapid development in enhancing pre-trained language models (PLMs) with various external knowledge bases, i.e., knowl- edge injection for PLMs (Levine et al., 2020; Zhou et al., 2020; Zhang et al., 2019; Peters et al., 2019; ∗ Equal contribution † Corresponding authors Figure 1: Illustration of plug-and-play knowledge in- jection, where knowledge bases and models are decou- pled. Bosselut et al., 2019; Guan et al., 2020). Knowl- edge injection improves the performance of PLMs on a wide range of tasks such as information extrac- tion (Liu et al., 2020a; Wang et al., 2021b), ques- tion answering (Xiong et al., 2020; Wang et al., 2021a), and text generation (Chen et al., 2020). Existing injection methods commonly inject knowledge by knowledge-aware pre-training or fine-tuning (Peters et al., 2019; Yamada et al., 2020; Liu et al., 2020a; Wang et al., 2021a). However, rarely studied is how to inject knowledge into a downstream model that is already adapted to a spe- cific task. If we want to apply a new knowledge injection method to enhance models on a specific task, we have to discard task-specific downstream models and retrain them. In addition, one down- stream model working with multiple knowledge bases requires retraining itself to inject each knowl- edge base. Retraining models is time-consuming and resource-intensive, leading to the need for a flexible and efficient injection paradigm. Toward flexible and efficient injection, …

Content truncated to 500 words