Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Models

Findings of the Association for Computational Linguistics: ACL 2022, pages 2824 - 2835 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Models Robert L. Logan IV1 Ivana Balaževi´c∗2 Eric Wallace3 Fabio Petroni4 Sameer Singh1 Sebastian Riedel4,5 1UC Irvine 2DeepMind 3UC Berkeley 4Facebook AI Research {rlogan,sameer}@uci.edu5University College London balazevic@deepmind.com ericwallace@berkeley.edu {fabiopetroni,sriedel}@fb.com Abstract Prompting language models (LMs) with train- ing examples and task descriptions has been seen as critical to recent successes in few-shot learning. In this work, we show that finetun- ing LMs in the few-shot setting can consid- erably reduce the need for prompt engineer- ing. In fact, one can use null prompts, prompts that contain neither task-specific templates nor training examples, and achieve competitive ac- curacy to manually-tuned prompts across a wide range of tasks. While finetuning LMs does introduce new parameters for each down- stream task, we show that this memory over- head can be substantially reduced—finetuning only the bias terms can achieve comparable or better accuracy than standard finetuning while only updating 0.1% of the parameters. All in all, we recommend finetuning LMs for few-shot learning as it is more accurate, has relatively stable performance across different prompts, and can be made nearly as efficient as using frozen LMs. 1 Introduction Few-shot learning—the ability to learn tasks with limited examples—is an important academic and practical challenge (Lake et al., 2015). In state- of-the-art NLP, few-shot learning is performed by reformulating tasks as natural language “prompts” and completing those prompts with pre-trained lan- guage models (Brown et al., 2020; Schick and Schütze, 2021a). Prompts that are well-designed can substantially improve accuracy (Zhao et al., 2021; Lu et al., 2021). However, finding these prompts is difficult: it requires a non-trivial combi- natorial search over the prompt’s wording (a.k.a. its pattern or template), whether and how to include training examples, and how to convert language model probabilities into class predictions. Conse- quently, prompts are often designed using human ∗Work done while an intern at Facebook AI Research. intuition that is hard to replicate and apply in a principled manner (Perez et al., 2021). In this work, we seek to mitigate prompt engi- neering by identifying a class of simple prompts that are effective across many tasks for masked language models (LMs). We find that, when us- ing prompt-based finetuning (Schick and Schütze, 2021a; Gao et al., 2021), the prompt requires less optimization than previously thought; in fact, the pattern and training examples can be com- pletely cut out (e.g., Figure 1, right). These null prompts—simple concatenations of the inputs and the [MASK] token—achieve comparable accuracy to manually-written patterns while drastically sim- plifying prompt design: users only need to decide the label names (a.k.a. the verbalizer) and where to place the [MASK] token. The effectiveness of null prompts also challenges the common wisdom that the success of few-shot learning is due to inductive biases present in the prompt. A key drawback of prompt-based finetuning is that it has large memory …

Content truncated to 500 words