K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1405–1418 August 1–6, 2021. ©2021 Association for Computational Linguistics 1405 K-ADAPTER: Infusing Knowledge into Pre-Trained Models with Adapters Ruize Wang1∗, Duyu Tang2, Nan Duan2, Zhongyu Wei1, Xuanjing Huang1, Jianshu Ji3, Guihong Cao3, Daxin Jiang2, Ming Zhou2 1Fudan University, Shanghai, China 2Microsoft, Beijing, China 3Microsoft, Redmond WA, USA {rzwang18,zywei,xjhuang}@fudan.edu.cn {dutang,nanduan,jianshuj,gucao,djiang,mingzhou}@microsoft.com Abstract We study the problem of injecting knowl- edge into large pre-trained models like BERT and RoBERTa. Existing methods typically update the original parameters of pre-trained models when injecting knowledge. However, when multiple kinds of knowledge are injected, the historically injected knowledge would be flushed away. To address this, we propose K- ADAPTER, a framework that retains the origi- nal parameters of the pre-trained model fixed and supports the development of versatile knowledge-infused model. Taking RoBERTa as the backbone model, K-ADAPTER has a neural adapter for each kind of infused knowl- edge, like a plug-in connected to RoBERTa. There is no information flow between differ- ent adapters, thus multiple adapters can be efficiently trained in a distributed way. As a case study, we inject two kinds of knowl- edge in this work, including (1) factual knowl- edge obtained from automatically aligned text- triplets on Wikipedia and Wikidata and (2) linguistic knowledge obtained via dependency parsing. Results on three knowledge-driven tasks, including relation classification, entity typing, and question answering, demonstrate that each adapter improves the performance and the combination of both adapters brings further improvements. Further analysis in- dicates that K-ADAPTER captures versatile knowledge than RoBERTa. 1 1 Introduction Language representation models, which are pre- trained on large-scale text corpus through unsu- pervised objectives like (masked) language mod- eling, such as BERT (Devlin et al., 2019), GPT (Radford et al., 2018, 2019), XLNet (Yang et al., ∗Work is done during internship at Microsoft. Zhongyu Wei and Duyu Tang are corresponding authors. 1Codes are publicly available at https://github. com/microsoft/K-Adapter 2019), RoBERTa (Liu et al., 2019) and T5 (Raf- fel et al., 2020), have established state-of-the-art performances on various NLP downstream tasks. Despite the huge success of these pre-trained mod- els in empirical studies, recent studies suggest that models learned in such an unsupervised manner struggle to capture rich knowledge. For exam- ple, Poerner et al. (2020) suggest that although language models do well in reasoning about the sur- face form of entity names, they fail in capturing rich factual knowledge. Kassner and Sch¨utze (2020) ob- serve that BERT mostly did not learn the meaning of negation (e.g. “not”). These observations mo- tivate us to study the injection of knowledge into pre-trained models like BERT and RoBERTa. Recently, some efforts have been made to ex- ploit injecting knowledge into pre-trained language models (Zhang et al., 2019; Lauscher et al., 2019; Levine et al., 2020; Peters et al., 2019; He et al., 2020; Xiong et al., 2020). Most previous works (as shown in Table 1) augment the standard lan- guage modeling objective with knowledge-driven objectives and update the entire model parameters. Although these methods obtain …

Content truncated to 500 words