Knowledge Prompting in Pre-trained Language Model for Natural Language Understanding

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3164 - 3177 December 7-11, 2022 ©2022 Association for Computational Linguistics Knowledge Prompting in Pre-trained Language Model for Natural Language Understanding Jianing Wang1, Wenkang Huang2, Qiuhui Shi2, Hongbin Wang2, Minghui Qiu3, Xiang Li1∗, Ming Gao1,4 1 School of Data Science and Engineering, East China Normal University, Shanghai, China 2 Ant Group, Hangzhou, China 3 Alibaba Group, Hangzhou, China 4 KLATASDS-MOE, School of Statistics, East China Normal University, Shanghai, China lygwjn@gmail.com, wenkang.hwk@alibaba-inc.com {qiuhui.sqh,hongbin.whb}@antgroup.com minghui.qmh@alibaba-inc.com {xiangli,mgao}@dase.ecnu.edu.cn Abstract Knowledge-enhanced Pre-trained Language Model (PLM) has recently received significant attention, which aims to incorporate factual knowledge into PLMs. However, most ex- isting methods modify the internal structures of fixed types of PLMs by stacking compli- cated modules, and introduce redundant and irrelevant factual knowledge from knowledge bases (KBs). In this paper, to address these problems, we introduce a seminal knowledge prompting paradigm and further propose a knowledge-prompting-based PLM framework KP-PLM. This framework can be flexibly com- bined with existing mainstream PLMs. Specifi- cally, we first construct a knowledge sub-graph from KBs for each context. Then we design multiple continuous prompts rules and trans- form the knowledge sub-graph into natural lan- guage prompts. To further leverage the factual knowledge from these prompts, we propose two novel knowledge-aware self-supervised tasks including prompt relevance inspection and masked prompt modeling. Extensive ex- periments on multiple natural language under- standing (NLU) tasks show the superiority of KP-PLM over other state-of-the-art methods in both full-resource and low-resource settings 1. 1 Introduction Pre-trained Language Models (PLMs) have be- come the dominant infrastructure for a majority of downstream natural language understanding (NLU) tasks, which generally adopt a two-stage training strategy, i.e., pre-training and fine-tuning. Recent notable PLMs include BERT (Devlin et al., 2019), ∗ Correspondence to Xiang Li. 1All the codes and datasets have been released to https: //github.com/wjn1996/KP-PLM Masked Mention Deep Transformer Encoder MLM Head Mention Deep Transformer Encoder Fusion Module Knowledge Encoder KG Entity Word-Knowledge Graph Graph Neural Networks MLM Head [SEP] Deep Transformer Encoder MLM Head Sub-graph Mention1 Mention2 Continuous Prompting Prompt1 Prompt2 Knowledge Head (a) Knowledge Masking (b) Knowledge Fusion (c) Graph Learning (d) Our Proposal: Knowledge Prompting PLM Module New Module Special Token Masked Token Entity Mention Token [CLS] [SEP] [CLS] [SEP] [CLS] [CLS] [SEP] [SEP] Figure 1: The comparison between knowledge prompt- ing with other paradigms. (Best viewed in color.) RoBERTa (Liu et al., 2019), ALBERT (Lan et al., 2020), XLNet (Yang et al., 2019), GPT-2 (Radford et al., 2019) and DeBERTa (He et al., 2021b). Al- though many existing models learn useful inherent linguistic knowledge from large-scale corpora, it is hard for them to understand the explicit factual knowledge (IV et al., 2019; Sun et al., 2020). To further leverage factual knowledge, a branch of knowledge-enhanced methods (Zhang et al., 2019; Sun et al., 2020; He et al., 2021a; Wang et al., 2021b,a; Zhang et al., 2021; Arora et al., 2022) have been proposed for PLMs to capture rich semantic knowledge from knowledge …

Content truncated to 500 words