Pre-Training to Learn in Context

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 4849–4870 July 9-14, 2023 ©2023 Association for Computational Linguistics Pre-Training to Learn in Context Yuxian Gu1,2,∗, Li Dong2, Furu Wei2, Minlie Huang1,† 1The CoAI Group, DCST, Institute for Artificial Intelligence, State Key Lab of Intelligent Technology and Systems, Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing 100084, China 2 Microsoft Research guyx21@mails.tsinghua.edu.cn, {lidong1,fuwei}@microsoft.com aihuang@tsinghua.edu.cn Abstract In-context learning, where pre-trained language models learn to perform tasks from task ex- amples and instructions in their contexts, has attracted much attention in the NLP commu- nity. However, the ability of in-context learn- ing is not fully exploited because language models are not explicitly trained to learn in context. To this end, we propose PICL (Pre- training for In-Context Learning), a framework to enhance the language models’ in-context learning ability by pre-training the model on a large collection of “intrinsic tasks” in the general plain-text corpus using the simple lan- guage modeling objective. PICL encourages the model to infer and perform tasks by con- ditioning on the contexts while maintaining task generalization of pre-trained models. We evaluate the in-context learning performance of the model trained with PICL on seven widely- used text classification datasets and the SUPER- NATURALINSTRCTIONS benchmark, which contains 100+ NLP tasks formulated to text generation. Our experiments show that PICL is more effective and task-generalizable than a range of baselines, outperforming larger lan- guage models with nearly 4x parameters. The code is publicly available at https://github. com/thu-coai/PICL. 1 Introduction Pre-trained language models (PLMs; Han et al., 2021; Qiu et al., 2020) have shown strong abili- ties of learning and performing unseen tasks con- ditioning on several task examples or instructions in its context, which is called in-context learning (ICL; Brown et al., 2020). Compared to conven- tional fine-tuning methods, ICL adapts PLMs to downstream tasks only through inference, with- out parameter updates, which is computationally cheaper in practice and is closer to general AI. † Corresponding author. ∗ Contribution during internship at Microsoft Research. However, PLMs trained on massive corpora to predict the next word given previous words are not explicitly taught to learn in the context. This makes ICL a surprising emergent ability but also indicates that the ICL ability of PLMs is not fully exploited. Garg et al. (2022) has shown that by directly train- ing to do ICL in a meta-learning paradigm, models show strong performance on learning simple func- tion classes in the context. In practical NLP scenar- ios, previous works (Min et al., 2022b; Chen et al., 2022b) also enhance the ICL performance by meta- fine-tuning PLMs on a large collection of down- stream tasks and evaluating them on unseen tasks. However, the low diversity of human-annotated downstream tasks restricts the performance of the meta-tuned model. Direct training on downstream tasks also brings undesired bias on specific input formats, label spaces, or domains, which hurts the generalization of PLMs. To enhance the ICL ability while …

Content truncated to 500 words