Improving In-Context Few-Shot Learning via Self-Supervised Training

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3558 - 3573 July 10-15, 2022 ©2022 Association for Computational Linguistics Improving In-Context Few-Shot Learning via Self-Supervised Training Mingda Chen1∗ Jingfei Du2 Ramakanth Pasunuru2 Todor Mihaylov2 Srini Iyer2 Veselin Stoyanov2 Zornitsa Kozareva2 1Toyota Technological Institute at Chicago, IL, USA 2Meta AI mchen@ttic.edu {jingfeidu,rpasunuru,tbmihaylov,sviyer,ves,zori}@fb.com Abstract Self-supervised pretraining has made few-shot learning possible for many NLP tasks. But the pretraining objectives are not typically adapted specifically for in-context few-shot learning. In this paper, we propose to use self- supervision in an intermediate training stage between pretraining and downstream few-shot usage with the goal to teach the model to per- form in-context few shot learning. We pro- pose and evaluate four self-supervised objec- tives on two benchmarks. We find that the intermediate self-supervision stage produces models that outperform strong baselines. Ab- lation study shows that several factors af- fect the downstream performance, such as the amount of training data and the diversity of the self-supervised objectives. Human-annotated cross-task supervision and self-supervision are complementary. Qualitative analysis suggests that the self-supervised-trained models are bet- ter at following task requirements. 1 Introduction In-context few-shot learning seeks to solve unseen tasks at inference time by conditioning on a few training examples. In particular, in this case we are interested in methods that forgo any weight up- dates (Brown et al., 2020). Prior work has been fo- cused on improving inference time algorithms (e.g., rescoring generated outputs (Zhao et al., 2021), se- lecting (Liu et al., 2021) and ordering (Lu et al., 2021) the given few-shot examples) and incorpo- rating extra resources (e.g., fine-tuning models on human-annotated datasets (Mishra et al., 2021; Ye et al., 2021; Wei et al., 2022)). We hypothesise that a different way to improve in-context few-shot learning is through designing self-supervised objectives that more closely resem- ble the format of tasks that the model will be asked to perform. To do so, we cast the self-supervised ∗Work done during an internship at Meta AI. training as an intermediate training stage between language model pretraining and downstream few- shot evaluation. In particular, we construct training datasets based on the self-supervised objectives following similar formats used in the downstream tasks, fine-tune pretrained language model check- points on the training datasets, and then evaluate the models on benchmarks. In experiments, we consider four self-supervised objectives, including masked word prediction and classification tasks related to next sentence pre- diction (Devlin et al., 2019). We evaluate mod- els on two benchmarks (13 tasks in total): Super- GLUE (Wang et al., 2019) and Natural-Instructions (Mishra et al., 2021). SuperGLUE focuses on dis- criminative tasks, and Natural-Instructions is a set of generative tasks. Empirically, we experiment with pretrained lan- guage models of two sizes: 125 million parameters and 1.3 billion parameters. We show that in our best setting, the 1.3 billion parameters model trained by the self-supervision performs better than the initial pretrained language models and two strong base- lines on …

Content truncated to 500 words