It’s Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2339–2352 June 6–11, 2021. ©2021 Association for Computational Linguistics 2339 It’s Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners Timo Schick1,2 and Hinrich Schütze1 1 Center for Information and Language Processing, LMU Munich, Germany 2 Sulzer GmbH, Munich, Germany timo.schick@sulzer.de Abstract When scaled to hundreds of billions of pa- rameters, pretrained language models such as GPT-3 (Brown et al., 2020) achieve remark- able few-shot performance. However, enor- mous amounts of compute are required for training and applying such big models, result- ing in a large carbon footprint and making it difficult for researchers and practitioners to use them. We show that performance similar to GPT-3 can be obtained with language mod- els that are much “greener” in that their pa- rameter count is several orders of magnitude smaller. This is achieved by converting textual inputs into cloze questions that contain a task description, combined with gradient-based op- timization; exploiting unlabeled data gives fur- ther improvements. We identify key factors re- quired for successful natural language under- standing with small language models.1 1 Introduction Pretraining ever-larger language models (LMs) on massive corpora has led to large improvements in NLP (Radford et al., 2018; Devlin et al., 2019; Liu et al., 2019; Raffel et al., 2020, i.a.). A standard approach is to replace the pretrained model’s out- put layer with a task-specific head and finetune the entire model on a set of labeled training data. However, language modeling is not only a pow- erful pretraining objective, but many tasks can be reformulated as cloze questions (e.g., by append- ing phrases such as “the correct answer is __”), allowing pretrained LMs to solve them without any or with only very few labeled examples (Radford et al., 2019; Schick and Schütze, 2021). Recently, Brown et al. (2020) introduced GPT-3, a pretrained LM with an enormous 175 billion pa- rameters, and showed that it has amazing few-shot abilities: By reformulating tasks as LM problems, 1Our implementation is publicly available at https:// github.com/timoschick/pet. 102 103 104 105 106 50 60 70 80 GPT-3 PET iPET Parameters (Millions) SuperGLUE Performance Figure 1: Performance on SuperGLUE with 32 train- ing examples. ALBERT with PET/iPET outperforms GPT-3 although it is much “greener” in that it has three orders of magnitude fewer parameters. GPT-3 achieves near state-of-the-art results for some SuperGLUE (Wang et al., 2019) tasks given just 32 labeled examples. This is achieved through priming: GPT-3 is given a few demonstrations of inputs and corresponding outputs as context for its predictions, but no gradient updates are performed. While being straightforward to use, this method has two major drawbacks: • It requires a gigantic LM to work well, mak- ing it unusable in many real-world scenar- ios and resulting in a large carbon footprint (Strubell et al., 2019). • It does not scale to more than a few exam- ples as the context window of most …

Content truncated to 500 words