Active Example Selection for In-Context Learning

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9134 - 9148 December 7-11, 2022 ©2022 Association for Computational Linguistics Active Example Selection for In-Context Learning Yiming Zhang and Shi Feng and Chenhao Tan {yimingz0, shif, chenhao}@uchicago.edu University of Chicago Abstract With a handful of demonstration examples, large-scale language models show strong ca- pability to perform various tasks by in-context learning from these examples, without any fine- tuning. We demonstrate that in-context learn- ing performance can be highly unstable across samples of examples, indicating the idiosyn- crasies of how language models acquire in- formation. We formulate example selection for in-context learning as a sequential decision problem, and propose a reinforcement learning algorithm for identifying generalizable policies to select demonstration examples. For GPT-2, our learned policies demonstrate strong abil- ities of generalizing to unseen tasks in train- ing, with a 5.8% improvement on average. Ex- amples selected from our learned policies can even achieve a small improvement on GPT-3 Ada. However, the improvement diminishes on larger GPT-3 models, suggesting emerging capabilities of large language models. 1 Introduction Large language models demonstrate the capability to learn from just a few examples (Radford et al., 2019; Brown et al., 2020; Rae et al., 2022; Zhang et al., 2022). The possibility to train a model with- out any parameter update has inspired excitement about the in-context learning paradigm. Intuitively, high in-context learning performance should require carefully chosen demonstration ex- amples, but a recent line of work suggests other- wise — that demonstration examples are not as important as we expected, and that few-shot perfor- mance can be largely attributed to the model’s zero- shot learning capacity (Min et al., 2022), across GPT-2 and GPT-3. This insight is corroborated by a parallel line of work that brings significant improvements to in-context learning performance without example selection, for example, by re- ordering randomly selected examples and using calibration (Lu et al., 2022; Zhao et al., 2021; Ko- jima et al., 2022). Another notable approach is to use best-of-n sampling, which requires a labeled set for validation (Nakano et al., 2022). Our contribution in this paper is twofold. First, we revisit the effect of example selection on in- context learning. We show that even with reorder- ing and calibration, we still observe a large variance across sets of demonstration examples, especially for GPT-2, while calibration reduces the variance for GPT-3 models. The high variance needs further investigation, as we take it as evidence that large language models are still not capable of efficiently and reliably acquire new information in-context. Understanding what makes good demonstration ex- amples sheds some light on the mechanisms that large language models use to process information. Second, we seek to discover general trends in example selection for in-context learning across different tasks. Concretely, we use reinforcement learning to optimize example selection as sequen- tial decision making problem. We argue that active example selection from unlabeled datasets is the most appropriate setting for in-context learning be- cause fine-tuning with an …

Content truncated to 500 words