Revisiting Self-training for Few-shot Learning of Language Model

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9125–9135 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 9125 Revisiting Self-Training for Few-Shot Learning of Language Model Yiming Chen†,‡ Yan Zhang† Chen Zhang† Grandee Lee† Ran Cheng‡,∗ Haizhou Li†,⋆,⋆⋆ †National University of Singapore ‡Southern University of Science and Technology ⋆The Chinese University of Hong Kong (Shenzhen) ⋆⋆Kriston AI Lab, China {yiming.chen,chen_zhang,grandee.lee}@u.nus.edu, {haizhou.li,eleyanz}@nus.edu.sg, ranchengcn@gmail.com Abstract As unlabeled data carry rich task-relevant in- formation, they are proven useful for few- shot learning of language model. The ques- tion is how to effectively make use of such data. In this work, we revisit the self-training technique for language model fine-tuning and present a state-of-the-art prompt-based few- shot learner, SFLM. Given two views of a text sample via weak and strong augmentation techniques, SFLM generates a pseudo label on the weakly augmented version. Then, the model predicts the same pseudo label when fine-tuned with the strongly augmented ver- sion. This simple approach is shown to out- perform other state-of-the-art supervised and semi-supervised counterparts on six sentence classification and six sentence-pair classifica- tion benchmarking tasks. In addition, SFLM only relies on a few in-domain unlabeled data. We conduct a comprehensive analysis to demonstrate the robustness of our proposed ap- proach under various settings, including aug- mentation techniques, model scale, and few- shot knowledge transfer across tasks. 1 1 Introduction Pre-trained language models (Devlin et al., 2019; Liu et al., 2019; Radford et al., 2019; Yang et al., 2019; Lan et al., 2020; Raffel et al., 2020; Clark et al., 2020) have set new state-of-the-art perfor- mance in many downstream NLP tasks. However, such performance often relies on large-scale high- quality supervision. Unfortunately, labeled data are not always available in practice. Recently, Brown et al. (2020) study how to fa- cilitate the few-shot learning of language models via the GPT-3 model. It achieves remarkable per- formance on many NLP datasets without any gradi- ent updates, by incorporating task-specific prompts ∗ Corresponding author. 1Our code is publicly available at https://github. com/MatthewCYM/SFLM into the text and reformulating the task as language modeling problems. However, GPT-3 has 175B parameters, that has a footprint too large for many real-world applications. Gao et al. (2020) applies the concept of prompt strategies in GPT-3 to small- footprint language models, such as BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019). After fine-tuned on a few annotated samples, the small- footprint models exhibit comparable performance to that of the large-footprint GPT-3 model. How- ever, the performance of these models is still lag- ging behind those under supervised learning, which have a much smaller footprint. Intuitively, unla- beled data also carry rich information of down- stream tasks and are more available than labelled data. In this paper, we focus on the few-shot learn- ing of language model with a small amount of la- beled and unlabeled data. Semi-supervised learning benefits from partially labeled datasets. A common implementation of semi-supervised learning is self-training, which leverages supervision signals offered by …

Content truncated to 500 words