Pre-trained Language Models Can be Fully Zero-Shot Learners

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 15590–15606 July 9-14, 2023 ©2023 Association for Computational Linguistics Pre-trained Language Models Can be Fully Zero-Shot Learners Xuandong Zhao†, Siqi Ouyang†, Zhiguo Yu‡, Ming Wu‡, Lei Li† †UC Santa Barbara ‡Microsoft {xuandongzhao,siqiouyang,leili}@cs.ucsb.edu {zhiguo.yu,mingwu}@microsoft.com Abstract How can we extend a pre-trained model to many language understanding tasks, without la- beled or additional unlabeled data? Pre-trained language models (PLMs) have been effective for a wide range of NLP tasks. However, existing approaches either require fine-tuning on downstream labeled datasets or manually constructing proper prompts. In this paper, we propose nonparametric prompting PLM (NPPrompt) for fully zero-shot language under- standing. Unlike previous methods, NPPrompt uses only pre-trained language models and does not require any labeled data or additional raw corpus for further fine-tuning, nor does it rely on humans to construct a comprehen- sive set of prompt label words. We eval- uate NPPrompt against previous major few- shot and zero-shot learning methods on di- verse NLP tasks: text classification, text entail- ment, similar text retrieval, paraphrasing, and multiple-choice question answering. Experi- mental results demonstrate that our NPPrompt outperforms the previous best fully zero-shot method by big margins, with absolute gains of 12.8% in accuracy on text classification and 15.6% on the GLUE benchmark. Our source code is available at https://github. com/XuandongZhao/NPPrompt. 1 Introduction Natural language understanding (NLU) has been important in many applications such as intelligent dialog assistants, online search, and social media analysis. Recent advancement of NLU has been driven by emergent pre-trained language models (PLMs) including BERT (Devlin et al., 2019; Liu et al., 2019b), GPT (Radford et al., 2018, 2019; Brown et al., 2020), BART (Lewis et al., 2020), and T5 (Raffel et al., 2020). Prior studies show that PLMs obtain substantial knowledge during pre- training on raw text corpus (Petroni et al., 2019; Feldman et al., 2019). By fine-tuning on task- specific labeled data, PLMs exploit such knowl- edge and gain impressive accuracy on a wide range of NLP tasks, such as text classification (Kowsari et al., 2019), question answering (Rajpurkar et al., 2016), machine reading comprehension (Campos et al., 2016), etc. However, fine-tuning approaches are expensive. It requires labeled datasets, which are rarely avail- able for many tasks. Significant computational efforts are needed to update PLMs’ parameters for multiple tasks. In addition, fine-tuning results in one distinct model for each task to maintain. How can we generalize a pre-trained model to many NLP tasks, without labeled or additional un- labeled data? Existing few-shot and zero-shot ap- proaches propose to construct prompts to elicit de- sired predictions from PLMs (Brown et al., 2020). The main idea of prompting PLMs is to convert an input utterance to one with masked templates. For example, in text classification an input can be “The Warriors won the NBA championship 2022” and it is instead converted to “The Warriors won the NBA championship 2022. This topic is about [MASK]”. A PLM (e.g. BERT) takes the converted text …

Content truncated to 500 words