Making Pre-trained Language Models Better Few-shot Learners

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 3816–3830 August 1–6, 2021. ©2021 Association for Computational Linguistics 3816 Making Pre-trained Language Models Better Few-shot Learners Tianyu Gao†∗ Adam Fisch‡∗ Danqi Chen† †Princeton University ‡Massachusetts Institute of Technology {tianyug,danqic}@cs.princeton.edu fisch@csail.mit.edu Abstract The recent GPT-3 model (Brown et al., 2020) achieves remarkable few-shot perfor- mance solely by leveraging a natural-language prompt and a few task demonstrations as in- put context. Inspired by their findings, we study few-shot learning in a more practical sce- nario, where we use smaller language models for which fine-tuning is computationally effi- cient. We present LM-BFF—better few-shot fine-tuning of language models1—a suite of simple and complementary techniques for fine- tuning language models on a small number of annotated examples. Our approach includes (1) prompt-based fine-tuning together with a novel pipeline for automating prompt genera- tion; and (2) a refined strategy for dynamically and selectively incorporating demonstrations into each context. Finally, we present a sys- tematic evaluation for analyzing few-shot per- formance on a range of NLP tasks, including classification and regression. Our experiments demonstrate that our methods combine to dra- matically outperform standard fine-tuning pro- cedures in this low resource setting, achieving up to 30% absolute improvement, and 11% on average across all tasks. Our approach makes minimal assumptions on task resources and do- main expertise, and hence constitutes a strong task-agnostic method for few-shot learning.2 1 Introduction The GPT-3 model (Brown et al., 2020) has made waves in the NLP community by demonstrating as- tounding few-shot capabilities on myriad language understanding tasks. Given only a natural lan- guage prompt and a few demonstrations of the task, GPT-3 is able to make accurate predictions without updating any of the weights of its underlying lan- *The first two authors contributed equally. 1Alternatively, language models’ best friends forever. 2Our implementation is publicly available at https:// github.com/princeton-nlp/LM-BFF. guage model. However, while remarkable, GPT-3 consists of 175B parameters, which makes it chal- lenging to use in most real-wold applications. In this work, we study a more practical scenario in which we only assume access to a moderately- sized language model such as BERT (Devlin et al., 2019) or RoBERTa (Liu et al., 2019), and a small number of examples (i.e., a few-shot setting), which we can use to fine-tune the weights of the language model. This setting is appealing as (1) such mod- els can be trained on typical research hardware; (2) few-shot settings are realistic, as it is generally both easy to acquire a few annotations (e.g., 32 examples) and efficient to train on them; and (3) updating parameters typically leads to better perfor- mance. Inspired by GPT-3’s findings, we propose several novel strategies for expanding its few-shot learning abilities to our setting, considering both classification and—for the first time—regression. First, we follow the route of prompt-based pre- diction, first developed by the GPT series (Radford et al., 2018, 2019; Brown et al., 2020) for zero-shot …

Content truncated to 500 words