STraTA: Self-Training with Task Augmentation for Better Few-shot Learning

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5715–5731 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 5715 STraTA: Self-Training with Task Augmentation for Better Few-shot Learning Tu Vu1,2⋆, Minh-Thang Luong2, Quoc V. Le2, Grady Simon2, Mohit Iyyer1 University of Massachusetts Amherst1 Google Research2 {tuvu,miyyer}@cs.umass.edu {ttvu,thangluong,qvl,gradys}@google.com Abstract Despite their recent successes in tackling many NLP tasks, large-scale pre-trained language models do not perform as well in few-shot set- tings where only a handful of training exam- ples are available. To address this shortcom- ing, we propose STraTA, which stands for Self-Training with Task Augmentation, an ap- proach that builds on two key ideas for effec- tive leverage of unlabeled data. First, STraTA uses task augmentation, a novel technique that synthesizes a large amount of data for auxiliary-task fine-tuning from target-task un- labeled texts. Second, STraTA performs self- training by further fine-tuning the strong base model created by task augmentation on a broad distribution of pseudo-labeled data. Our exper- iments demonstrate that STraTA can substan- tially improve sample efficiency across 12 few- shot benchmarks. Remarkably, on the SST-2 sentiment dataset, STraTA, with only 8 train- ing examples per class, achieves comparable results to standard fine-tuning with 67K train- ing examples. Our analyses reveal that task augmentation and self-training are both com- plementary and independently effective. 1 Introduction Recent advances in NLP demonstrate the effec- tiveness of applying large-scale pretrained lan- guage models to downstream tasks (Devlin et al., 2019; Liu et al., 2019; Yang et al., 2019; Lan et al., 2020; Raffel et al., 2020; Brown et al., 2020; He et al., 2021). While these models have achieved state-of-the-art results on many NLP benchmarks, they struggle when given limited train- ing data. For instance, Devlin et al. (2019) find that BERT is prone to degenerate performance on small datasets. While enormous language models like GPT-3 (Brown et al., 2020) exhibit the ability to solve a new task from only a few examples without ⋆ Work done as a student researcher at Google Brain. # labeled examples per class Accuracy Accuracy # labeled examples per class Accuracy Figure 1: Our Self-Training with Task Augmentation (STraTA) approach substantially improves sample effi- ciency across different tasks. For example, when given only 8 labeled examples per class from the SST-2 sen- timent dataset, STraTA is competitive with standard fine-tuning on 67K examples; on the SciTail entailment dataset, with 512 labeled examples per class, STraTA surpasses standard fine-tuning on 27K examples. any fine-tuning, their performance still lags far be- hind state-of-the-art fine-tuning results. Manually annotating large amounts of training data will likely improve performance but can also be prohibitively expensive to obtain for many tasks and domains. In this paper, we propose STraTA, an approach that combines two complementary methods, Self- Training and Task Augmentation, to effectively leverage unlabeled data, which is comparatively cheaper to obtain.1 At a high level, task augmentation exploits un- labeled text from the domain of a given target task to simulate a large amount of in-domain …

Content truncated to 500 words