Self-training Improves Pre-training for Natural Language Understanding

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5408–5418 June 6–11, 2021. ©2021 Association for Computational Linguistics 5408 Self-training Improves Pre-training for Natural Language Understanding Jingfei Du†∗ Edouard Grave† Beliz Gunel‡ Vishrav Chaudhary† Onur Celebi† Michael Auli† Veselin Stoyanov† Alexis Conneau†∗ {jingfeidu,egrave,vishrav,celebi,michaelauli,ves,aconneau}@fb.com, bgunel@stanford.edu †Facebook AI, ‡Stanford University Abstract Unsupervised pre-training has led to much re- cent progress in natural language understand- ing. In this paper, we study self-training as an- other way to leverage unlabeled data through semi-supervised learning. To obtain addi- tional data for a specific task, we introduce SentAugment, a data augmentation method which computes task-specific query embed- dings from labeled data to retrieve sentences from a bank of billions of unlabeled sentences crawled from the web. Unlike previous semi- supervised methods, our approach does not re- quire in-domain unlabeled data and is there- fore more generally applicable. Experiments show that self-training is complementary to strong RoBERTa baselines on a variety of tasks. Our augmentation approach leads to scalable and effective self-training with im- provements of up to 2.6% on standard text classification benchmarks. Finally, we also show strong gains on knowledge-distillation and few-shot learning. 1 Introduction Self-training is a semi-supervised method which uses a teacher model, trained using labeled data, to create synthetic labels for unlabeled exam- ples (Scudder, 1965; Yarowsky, 1995). These syn- thetic labels are then used to train a student model. This approach is called self-training when the stu- dent model has a similar or higher capacity than the teacher, and knowledge distillation (Hinton et al., 2015) when the student model is smaller than the teacher. Self-training has been successfully ap- plied to a variety of tasks, including image recog- nition (Yalniz et al., 2019; Xie et al., 2020; Zoph et al., 2020), automatic speech recognition (Syn- naeve et al., 2019; Kahn et al., 2020; Park et al., 2020), sequence generation (He et al., 2019), and parsing (McClosky et al., 2006). ∗Equal contribution. An alternative semi-supervised technique is pre- training (Dai and Le, 2015; Radford et al., 2018; Howard and Ruder, 2018; Devlin et al., 2018), which has led to large improvements for natural language understanding compared to purely super- vised learning. In that case, models are first trained on an auxiliary task, such as language modeling, followed by fine-tuning on the task of interest. A natural question is the following: do pre- training and self-training capture the same infor- mation, or are they complementary? Recently, Zoph et al. (2020) studied this question in the context of image recognition, showing that self- training was helpful, even in addition to pre- training. However, their study mostly considers su- pervised pre-training, in which models were trained on ImageNet classification. Moreover, in cases where large amounts of supervised data were avail- able for the downstream task, pre-training was not helpful, even without self-training. This is in con- trast to natural language understanding for which language modeling pre-training is a very strong baseline that leads to …

Content truncated to 500 words