Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classification

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 2225 - 2240 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classification Shengding Hu1, Ning Ding1, Huadong Wang1∗, Zhiyuan Liu1,2,4∗, Jingang Wang3, Juanzi Li1, Wei Wu3, Maosong Sun1,2,4 1Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University, Beijing, China Beijing National Research Center for Information Science and Technology 2Institute Guo Qiang, Tsinghua University, Beijing, China 3Meituan, Beijing, China 4International Innovation Center of Tsinghua University, Shanghai, China {hsd20, dingn18}@mails.tsinghua.edu.cn Abstract Tuning pre-trained language models (PLMs) with task-specific prompts has been a promis- ing approach for text classification. Partic- ularly, previous studies suggest that prompt- tuning has remarkable superiority in the low- data scenario over the generic fine-tuning meth- ods with extra classifiers. The core idea of prompt-tuning is to insert text pieces, i.e., tem- plate, to the input and transform a classifica- tion problem into a masked language modeling problem, where a crucial step is to construct a projection, i.e., verbalizer, between a label space and a label word space. A verbalizer is usually handcrafted or searched by gradient descent, which may lack coverage and bring considerable bias and high variances to the re- sults. In this work, we focus on incorporating external knowledge into the verbalizer, form- ing a knowledgeable prompt-tuning (KPT), to improve and stabilize prompt-tuning. Specifi- cally, we expand the label word space of the ver- balizer using external knowledge bases (KBs) and refine the expanded label word space with the PLM itself before predicting with the ex- panded label word space. Extensive experi- ments on zero and few-shot text classification tasks demonstrate the effectiveness of knowl- edgeable prompt-tuning. Our source code is publicly available at https://github.com/ thunlp/KnowledgeablePromptTuning. 1 Introduction Recent years have witnessed the prominence of Pre- trained Language Models (PLMs) (Peters et al., 2018; Radford et al., 2018; Devlin et al., 2019; Raffel et al., 2020; Xu et al., 2021) due to their superior performance on a wide range of language- related downstream tasks such as text classifica- tion (Kowsari et al., 2019), question answering (Ra- jpurkar et al., 2016), and machine reading compre- hension (Nguyen et al., 2016). To fathom the prin- ∗ Corresponding authors: Z.Liu (liuzy@tsinghua.edu.cn), H.Wang (huadw2012@163.com) ciples of such effectiveness of PLMs, researchers have conducted extensive studies and suggested that PLMs have obtained rich knowledge during pre-training (Petroni et al., 2019; Davison et al., 2019). Hence, how to stimulate and exploit such knowledge is receiving increasing attention. One conventional approach to achieve that is fine-tuning (Devlin et al., 2019), where we add extra classifiers on the top of PLMs and fur- ther train the models under classification objec- tives. Fine-tuning has achieved satisfying results on supervised tasks. However, since the extra classifier requires adequate training instances to tune, it is still challenging to apply fine-tuning in few-shot learning (Brown et al., 2020) and zero-shot learning (Yin et al., 2019) scenarios. Originated from GPT-3 (Brown …

Content truncated to 500 words