Improving and Simplifying Pattern Exploiting Training

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4980–4991 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 4980 Improving and Simplifying Pattern Exploiting Training Derek Tam∗ Rakesh R Menon∗ Mohit Bansal Shashank Srivastava Colin Raffel UNC Chapel Hill {dtredsox, rrmenon, mbansal, ssrivastava, craffel}@cs.unc.edu Abstract Recently, pre-trained language models (LMs) have achieved strong performance when fine- tuned on difficult benchmarks like Super- GLUE. However, performance can suffer when there are very few labeled examples available for fine-tuning. Pattern Exploiting Training (PET) is a recent approach that lever- ages patterns for few-shot learning. How- ever, PET uses task-specific unlabeled data. In this paper, we focus on few shot learn- ing without any unlabeled data and introduce ADAPET, which modifies PET’s objective to provide denser supervision during fine-tuning. As a result, ADAPET outperforms PET on Su- perGLUE without any task-specific unlabeled data. Our code can be found at https:// github.com/rrmenon10/ADAPET. 1 Introduction Pre-trained language models (LMs) have shown significant gains across a wide variety of natural language processing (NLP) tasks in recent years (Devlin et al., 2019; Radford et al., 2018; Raffel et al., 2020). Most of these gains are obtained by fine-tuning language models on labeled data for a particular task. However, performance can suffer when there is very limited labeled data available for a downstream task (Xie et al., 2020; Chen et al., 2020). Recently, GPT-3 (Brown et al., 2020) demon- strated how language models, when scaled to hun- dreds of billions of parameters, can learn well when primed with only a few labeled examples. How- ever, the scale of GPT-3 (175B parameters) makes it impractical to study. There is, therefore, a need to develop smaller language models that can work equally well with limited labeled data. Pattern-Exploiting Training (PET; Schick and Schütze, 2021a,b) reformulates natural language understanding tasks as cloze-style questions and ∗Equal contribution 10 102 103 104 105 70 72 74 76 78 ADAPET GPT-3 PET iPET 0.3% of the data Amount of task-specific data used SuperGLUE Performance Figure 1: Performance of ADAPET vs iPET/PET and GPT-3 on SuperGLUE. While iPET/PET are parameter- efficient, they use ∼9K unlabeled examples in addition to 32 labeled examples per task. ADAPET uses just 32 labeled examples, and performs better than iPET. performs gradient-based fine-tuning. In doing so, PET outperforms GPT-3 with few labeled examples using ALBERT (Lan et al., 2020). However, PET uses additional task-specific unlabeled data. We propose ADAPET (A Densely-supervised Approach to Pattern Exploiting Training) that uses more supervision by decoupling the losses for the label tokens and a label-conditioned masked lan- guage modeling (MLM) objective over the full orig- inal input. On SuperGLUE (Wang et al., 2019) with 32 labeled examples per task, ADAPET outperforms iPET without any unlabeled data. 2 Background Cloze-style questions and MLM. A cloze task is a problem where certain parts of a text are removed, and the goal is to replace the missing portion based on the context (Taylor, 1953). Here, the text that has some parts removed is …

Content truncated to 500 words