NSP-BERT: A Prompt-based Few-Shot Learner through an Original Pre-training Task —— Next Sentence Prediction

Proceedings of the 29th International Conference on Computational Linguistics, pages 3233–3250 October 12–17, 2022. 3233 NSP-BERT: A Prompt-based Few-Shot Learner Through an Original Pre-training Task —— Next Sentence Prediction Yi Sun∗, Yu Zheng∗, Chao Hao, Hangping Qiu† Army Engineering University of PLA, Nanjing, China sunyi_lgdx@sina.com, zhengyu87@outlook.com haochaoleo@163.com, qiuhp_zy@163.com Abstract Using prompts to utilize language models to perform various downstream tasks, also known as prompt-based learning or prompt- learning, has lately gained significant success in comparison to the pre-train and fine-tune paradigm. Nonetheless, virtually most prompt- based methods are token-level such as PET based on mask language model (MLM). In this paper, we attempt to accomplish several NLP tasks in the zero-shot and few-shot scenarios using a BERT original pre-training task aban- doned by RoBERTa and other models——Next Sentence Prediction (NSP). Unlike token-level techniques, our sentence-level prompt-based method NSP-BERT does not need to fix the length of the prompt or the position to be pre- dicted, allowing it to handle tasks such as entity linking with ease. NSP-BERT can be applied to a variety of tasks based on its properties. We present an NSP-tuning approach with binary cross-entropy loss for single-sentence classifi- cation tasks that is competitive compared to PET and EFL. By continuing to train BERT on RoBERTa’s corpus, the model’s performance improved significantly, which indicates that the pre-training corpus is another important de- terminant of few-shot besides model size and prompt method.1 1 Introduction GPT-2 (up to 1.5B (Radford et al., 2019)) and GPT- 3 (up to 175B (Brown et al., 2020)) are ultra-large- scale language models with billions of parameters that have recently demonstrated outstanding per- formance in various NLP tasks. Compared with previous state-of-the-art fine-tuning methods, they can achieve competitive results without any or with just a limited quantity of training data. Although ∗Equal contribution †Corresponding author 1Our code and pre-trained models are publicly at: https: //github.com/sunyilgdx/Prompts4Keras. [CLS] [EOS] Franklin drafted the Declaration of Independence. NSP [SEP] The Founding Fathers of US. Franklin is Entity Linking [CLS] [EOS] Franklin drafted the Declaration of Independence. NSP [SEP] A Person Franklin is Person Entity Typing [CLS] [EOS] The phone on the desk rang. It hung up after two minutes. NSP [SEP] phone It means Coreference Resolution [EOS] The screen stopped working. [SEP] [CLS] NSP I’am angry. Negative [CLS] [EOS] Italian won the European Cup. NSP [SEP] It’s sports new. Sports Sentence Classification [CLS] [EOS] Every [BLANK] has two side. NSP [SEP] coin Cloze-style [CLS] [EOS] Countdown 2 minutes. NSP [SEP] Remind me in two minutes. Sentence Pair Classification Figure 1: Prompts for various NLP tasks of NSP-BERT. studies have shown that scaling up the model im- proves task-agnostic and few-shot performance, some studies have shown that by constructing ap- propriate prompts for the model, models like BERT (Devlin et al., 2018) or RoBERTa (Liu et al., 2019) can achieve similar performance despite having a parameter count that is several orders of magnitude smaller (Schick and Schütze, 2021b,a; Wang et al., 2021). Since then, the area of natural language pro- …

Content truncated to 500 words