True Few-Shot Learning with Prompts—A Real-World Perspective

True Few-Shot Learning with Prompts—A Real-World Perspective Timo Schick and Hinrich Sch¨utze Center for Information and Language Processing (CIS), LMU Munich, Germany schickt@cis.lmu.de, inquiries@cislmu.org Abstract Prompt-based approaches excel at few-shot learning. However, Perez et al. (2021) re- cently cast doubt on their performance as they had difficulty getting good results in a ‘‘true’’ few-shot setting in which prompts and hyperparameters cannot be tuned on a dev set. In view of this, we conduct an extensive study of PET, a method that combines textual instructions with example-based finetuning. We show that, if correctly configured, PET performs strongly in true few-shot settings without a dev set. Crucial for this strong perfor- mance is a number of design choices, including PET’s ability to intelligently handle multiple prompts. We put our findings to a real-world test by running PET on RAFT, a benchmark of tasks taken from realistic NLP applications for which no labeled dev or test sets are available. PET achieves a new state of the art on RAFT and performs close to non-expert humans for 7 out of 11 tasks. These results demonstrate that prompt-based learners can successfully be applied in true few-shot settings and underpin our belief that learning from instructions will play an important role on the path towards human-like few-shot learning capabilities. 1 Introduction With pretrained language models (LMs) getting ever larger (Radford et al., 2019; Raffel et al., 2020; Brown et al., 2020; Fedus et al., 2021), instruction-based learning is a powerful method for few-shot text classification (e.g., Schick and Sch¨utze, 2020; Jiang et al., 2020; Schick and Sch¨utze, 2021; Brown et al., 2020; Wei et al., 2022; Sanh et al., 2022). The key idea is to give an LM access to descriptive names for all possi- ble outputs and to short prompts explaining the task to be solved. In settings where at most a few dozen examples are available, this simple idea leads to substantial improvements over other ap- proaches (Schick and Sch¨utze, 2020, 2021; Gao et al., 2021; Tam et al., 2021). However, recent work has questioned the strong few-shot performance of instruction-based ap- proaches, arguing that they are evaluated in scenarios that are not true few-shot settings (Perez et al., 2021; Logan IV et al., 2021), mainly for two reasons. First, some approaches (e.g., Xie et al., 2019; Zhang et al., 2020; Chen et al., 2020; Tam et al., 2021) make use of large development sets to optimize hyperparameters. Second, it is argued that manually designed instructions require man- ual tuning on development sets to achieve strong performance (Perez et al., 2021; Logan IV et al., 2021). Indeed, performance can vary greatly—and in mostly unpredictable ways—across different in- structions (Jiang et al., 2020; Schick and Sch¨utze, 2020); this issue even persists after finetuning on hundreds of instructions (Sanh et al., 2022). More generally, the need for human involvement is seen as a serious drawback of manually designed in- structions (Shin et al., 2020; Lester et al., 2021). Thus, several recent studies have abandoned them in …

Content truncated to 500 words