Understanding In-Context Learning via Supportive Pretraining Data

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 12660–12673 July 9-14, 2023 ©2023 Association for Computational Linguistics Understanding In-Context Learning via Supportive Pretraining Data Xiaochuang Han♠♣∗ Dániel Simig♣ Todor Mihaylov♣ Yulia Tsvetkov♠ Asli Celikyilmaz♣ Tianlu Wang♣ ♣Meta AI ♠University of Washington {xhan77, yuliats}@cs.washington.edu simigd@gmail.com {tbmihaylov, aslic, tianluwang}@meta.com Abstract In-context learning (ICL) improves language models’ performance on a variety of NLP tasks by simply demonstrating a handful of examples at inference time. It is not well understood why ICL ability emerges, as the model has never been specifically trained on such demonstra- tions. Unlike prior work that explores implicit mechanisms behind ICL, we study ICL via in- vestigating the pretraining data. Specifically, we first adapt an iterative, gradient-based ap- proach to find a small subset of pretraining data that supports ICL. We observe that a continued pretraining on this small subset significantly im- proves the model’s ICL ability, by up to 18%. We then compare the supportive subset con- strastively with random subsets of pretraining data and discover: (1) The supportive pretrain- ing data to ICL do not have a higher domain rel- evance to downstream tasks. (2) The supportive pretraining data have a higher mass of rarely occurring, long-tail tokens. (3) The support- ive pretraining data are challenging examples where the information gain from long-range context is below average, indicating learning to incorporate difficult long-range context encour- ages ICL. Our work takes a first step towards understanding ICL via analyzing instance-level pretraining data. Our insights have a potential to enhance the ICL ability of language models by actively guiding the construction of pretrain- ing data in the future. 1 Introduction In-context learning in NLP has drawn tremendous attention recently (Dong et al., 2022). Unlike tra- ditional learning paradigms that rely on training or finetuning models, in-context learning only pro- vides a handful of demonstration examples to lan- guage models as a prefix to the test input, without any parameter updates. In-context learning has shown superior performance on a range of NLP tasks (Brown et al., 2020; Zhang et al., 2022b; ∗Work done during an internship at Meta AI. Pretraining data I can't think of any scenario where the Chiefs don't win that game if Charles doesn't go down. What's that? Need to chew clock with the run game? How convenient that we have an All Pro running back! While I agree that Charles going down definitely affected the outcome of the game, it's not like their back-up crapped the bed either. Knile Davis did end up with 2 TDs, so while he's not going to be mistaken for Charles, he played a great game ... In-context learning data Input: Microsoft may invest OpenAI if ... Category: Tech. Input: On Thursday, the White House said ... Category: Politics. Input: On Friday, Apple will introduce a new ... Category: [MASK] Figure 1: An example from the pretraining data of OPT (Zhang et al., 2022b) and an illustrative in-context learn- ing example of topic …

Content truncated to 500 words