Prompt-free and Efficient Few-shot Learning with Language Models

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 3638 - 3652 May 22-27, 2022 c⃝2022 Association for Computational Linguistics PERFECT: Prompt-free and Efficient Few-shot Learning with Language Models Rabeeh Karimi Mahabadi1,3,4 Luke Zettlemoyer1,2 James Henderson4 Marzieh Saeidi1 Lambert Mathias1 Veselin Stoyanov1 Majid Yazdani1 1Meta AI 2University of Washington 3EPFL 4Idiap Research Institute {rabeeh.karimi, james.henderson}@idiap.ch lsz@cs.washington.edu {marzieh,mathiasl,ves,myazdani}@fb.com Abstract Current methods for few-shot fine-tuning of pretrained masked language models (PLMs) require carefully engineered prompts and verbalizers for each new task to convert examples into a cloze-format that the PLM can score. In this work, we propose PERFECT, a simple and efficient method for few-shot fine-tuning of PLMs without relying on any such handcrafting, which is highly effective given as few as 32 data points. PERFECT makes two key design choices: First, we show that manually engineered task prompts can be replaced with task-specific adapters that enable sample-efficient fine-tuning and reduce memory and storage costs by roughly factors of 5 and 100, respectively. Second, instead of using handcrafted verbalizers, we learn new multi-token label embeddings during fine-tuning, which are not tied to the model vocabulary and which allow us to avoid complex auto-regressive decoding. These embeddings are not only learnable from limited data but also enable nearly 100x faster training and inference. Experiments on a wide range of few shot NLP tasks demon- strate that PERFECT, while being simple and efficient, also outperforms existing state-of-the- art few-shot learning methods. Our code is publicly available at https://github.com/ facebookresearch/perfect.git. 1 Introduction Recent methods for few-shot language model tuning obtain impressive performance but require careful engineering of prompts and verbalizers to convert inputs to a cloze-format (Taylor, 1953) that can be scored with pre-trained language models (PLMs) (Radford et al., 2018; Radford et al.; Brown et al., 2020; Schick and Schütze, 2021a,b). For example, as Figure 1 shows, a sentiment classifier can be designed by inserting the input text x in a prompt template “x It was [MASK]” where verbalizers (e.g., ‘great’ and ‘terrible’) are substituted for the [MASK] to score target task labels (‘positive’ or ‘negative’). In this paper, we show that such engineering is [CLS] The restaurant had excellent foods. It was [MASK] [SEP] Pretrained Language Model Input Pattern MLM Head terrible great Verbalizers positive negative Labels Figure 1: Existing few-shot fine-tuning methods require manual engineering to reduce new tasks to masked lan- guage modeling. PERFECT does not rely on any handcraft- ing, removing both patterns and verbalizers (see Figure 3). not needed for few-shot learning and instead can be replaced with simple methods for data-efficient fine-tuning with as few as 32 end-task examples. More specifically, we propose PERFECT, a Prompt-free and Efficient paRadigm for FEw-shot Cloze-based fine-Tuning. To remove handcrafted patterns, PERFECT uses task-specific adapter layers (Houlsby et al., 2019; Pfeiffer et al., 2020) (§3.1). Freezing the underlying PLM with millions or billions of parameters (Liu et al., 2019; Raffel et al., 2020), and only tuning adapters with very few new param- eters saves …

Content truncated to 500 words