Template-free Prompt Tuning for Few-shot NER

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5721 - 5732 July 10-15, 2022 ©2022 Association for Computational Linguistics Template-free Prompt Tuning for Few-shot NER Ruotian Ma1∗, Xin Zhou1∗, Tao Gui2†, Yiding Tan1, Linyang Li1, Qi Zhang1†, Xuanjing Huang1 1School of Computer Science, Fudan University, Shanghai, China 2Institute of Modern Languages and Linguistics, Fudan University, Shanghai, China {rtma19,xzhou20,tgui,qz,xjhuang}@fudan.edu.cn Abstract Prompt-based methods have been successfully applied in sentence-level few-shot learning tasks, mostly owing to the sophisticated design of templates and label words. However, when applied to token-level labeling tasks such as NER, it would be time-consuming to enumerate the template queries over all potential entity spans. In this work, we propose a more elegant method to reformulate NER tasks as LM prob- lems without any templates. Specifically, we discard the template construction process while maintaining the word prediction paradigm of pre-training models to predict a class-related pivot word (or label word) at the entity position. Meanwhile, we also explore principled ways to automatically search for appropriate label words that the pre-trained models can easily adapt to. While avoiding the complicated template-based process, the proposed LM objective also reduces the gap between different objectives used in pre-training and fine-tuning, thus it can better benefit the few-shot performance. Experimental results demonstrate the effectiveness of the proposed method over bert-tagger and template-based method under few-shot settings. Moreover, the decoding speed of the proposed method is up to 1930.12 times faster than the template-based method. 1 Introduction Pre-trained language models (LMs) have led to large improvements in NLP tasks (Devlin et al., 2019; Liu et al., 2019; Lewis et al., 2020). Popular practice to perform downstream classification tasks is to replace the pretrained model’s output layer with a classifier head and fine-tune it using a task-specific objective function. Recently, a new paradigm, prompt-based learning, has achieved great success on few-shot classification tasks by reformulating classification tasks as cloze ∗Equal contribution. †Corresponding authors. Obama is a [MASK] entity. Input LM predictions Obama was born in America . [CLS] [SEP] Obama was is a [MASK] entity. [CLS] Input [SEP] America . is a [MASK] entity. [CLS] Input [SEP] ...... person ➔ label: PER ✔ organization ➔ label: ORG location ➔ label: LOC none ➔ label: O Input: Query LM x 1 Query LM x 2 Query LM x 21 Figure 1: An example of template-based prompt method for NER. Predicting all labels in sentence “Obama was born in America." requires enumeration over all spans. questions. Typically, for each input [X], a template is used to convert [X] into an unfilled text (e.g., “[X] It was __."), allowing the model to fill in the blank with its language modeling ability. For instance, when performing sentiment classification task, the input “I love the milk." can be converted into “I love the milk. It was __.". Consequently, the LM may predict a label word “great", indicating that the input belongs to a positive class. Two main factors contribute …

Content truncated to 500 words