MetaICL: Learning to Learn In Context

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2791 - 2809 July 10-15, 2022 ©2022 Association for Computational Linguistics MetaICL: Learning to Learn In Context Sewon Min1,2 Mike Lewis2 Luke Zettlemoyer1,2 Hannaneh Hajishirzi1,3 1University of Washington 2Meta AI {sewon,lsz,hannaneh}@cs.washington.edu3Allen Institute for AI mikelewis@fb.com Abstract We introduce MetaICL (Meta-training for In- Context Learning), a new meta-training frame- work for few-shot learning where a pretrained language model is tuned to do in-context learn- ing on a large set of training tasks. This meta- training enables the model to more effectively learn a new task in context at test time, by sim- ply conditioning on a few training examples with no parameter updates or task-specific tem- plates. We experiment on a large, diverse col- lection of tasks consisting of 142 NLP datasets including classification, question answering, natural language inference, paraphrase detec- tion and more, across seven different meta- training/target splits. MetaICL outperforms a range of baselines including in-context learn- ing without meta-training and multi-task learn- ing followed by zero-shot transfer. We find that the gains are particularly significant for target tasks that have domain shifts from the meta-training tasks, and that using a diverse set of the meta-training tasks is key to im- provements. We also show that MetaICL approaches (and sometimes beats) the per- formance of models fully finetuned on the target task, and outperforms much bigger models with nearly 8x parameters. Finally, we show that MetaICL is complementary to human-written instructions, and the best per- formance can be achieved by combining both approaches. 1 Introduction Large language models (LMs) have recently been shown to be able to do in-context learning (Brown et al., 2020), where they learn a new task simply by conditioning on a few training examples and predicting which tokens best complete a test input. This type of learning is attractive because the model learns a new task through inference alone, without any parameter updates. However, performance sig- nificantly lags behind supervised finetuning, results are often high variance (Zhao et al., 2021; Perez et al., 2021), and it can be difficult to engineer the templates that convert existing tasks to this format. In this paper, we address these challenges by in- troducing MetaICL: Meta-training for In-Context Learning. MetaICL tunes a pretrained language model on a large set of tasks to learn how to in- context learn, and is evaluated on strictly new un- seen tasks. Each meta-training example matches the test setup—it includes k + 1 training examples from one task that will be presented together as a single sequence to the language model, and the output of the final example is used to calculate the cross-entropy training loss. Simply finetuning the model in this data setup directly leads to better in- context learning—the model learns to recover the semantics of the task from the given examples, as must be done for in-context learning of a new task at test time. This approach is related to recent …

Content truncated to 500 words