What Makes Pre-trained Language Models Better Zero-shot Learners?

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 2288–2303 July 9-14, 2023 ©2023 Association for Computational Linguistics What Makes Pre-trained Language Models Better Zero-shot Learners? Jinghui Lu1, Dongsheng Zhu+ 2, Weidong Han+ 2, Rui Zhao 1, Brian Mac Namee 3, Fei Tan∗ 1 1 SenseTime Research 2 Fudan University 3 School of Computer Science, University College Dublin {lujinghui1, zhaorui, tanfei}@sensetime.com {dszhu20, wdhan20}@fudan.edu.cn {brian.macnamee}@ucd.ie Abstract Current methods for prompt learning in zero- shot scenarios widely rely on a development set with sufficient human-annotated data to select the best-performing prompt template a poste- riori. This is not ideal because in a real-world zero-shot scenario of practical relevance, no labelled data is available. Thus, we propose a simple yet effective method for screening reasonable prompt templates in zero-shot text classification: Perplexity Selection (Perplec- tion). We hypothesize that language discrep- ancy can be used to measure the efficacy of prompt templates, and thereby develop a sub- stantiated perplexity-based scheme allowing for forecasting the performance of prompt tem- plates in advance. Experiments show that our method leads to improved prediction perfor- mance in a realistic zero-shot setting, eliminat- ing the need for any labelled examples. 1 Introduction Prompt learning has been demonstrated to be a successful remedy for challenges associated with pre-training and fine-tuning paradigm, especially in zero/few-shot scenarios (Gao et al., 2021; Schick and Schütze, 2021a,b; Tam et al., 2021; Lu et al., 2022a). Research has repeatedly shown that various transformer-based language models can benefit from prompt learning. For example, decoder-only models, such as those in the GPT family (Brown et al., 2020), can better generalise to unseen cases by prefixing inputs with a few training examples (in natural language). This is known as in-context learning (Brown et al., 2020; Xie et al., 2021; Liu et al., 2022a). Encoder-decoder models, such as T5 (Raffel et al., 2020) or BART (Lewis et al., 2020), can leverage prompt learning to train ver- satile models for multiple tasks (Khashabi et al., +Work was done during internship at SenseTime Research *Corresponding author 2020; Lester et al., 2021). Bidirectional encoder- only models, such as those in the BERT family (De- vlin et al., 2018; Liu et al., 2019), can also manifest impressive zero-shot capacity when given proper prompts. These prompts often take the form of pre-training tasks, such as next sentence predic- tion (Sun et al., 2022) or masked language model- ing (MLM) (Gao et al., 2021; Schick and Schütze, 2021a,b; Tam et al., 2021)—also known as cloze- style prompt learning. Despite its success in encoder-only models, cloze-style prompt learning is sensitive to the spe- cific involved templates. Multiple studies have shown that the design and choice of prompt tem- plates greatly affect the effectiveness of zero-shot learning (Tam et al., 2021; Zhao et al., 2021; Ru- bin et al., 2022). Ideally, they are supposed to be as close as possible to the language used in down- stream task. For example, in a sentiment analy- sis task, a suitable template …

Content truncated to 500 words