Z-ICL: Zero-Shot In-Context Learning with Pseudo-Demonstrations

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 2304–2317 July 9-14, 2023 ©2023 Association for Computational Linguistics Z-ICL: Zero-Shot In-Context Learning with Pseudo-Demonstrations Xinxi Lyu1 Sewon Min1 Iz Beltagy2 Luke Zettlemoyer1 Hannaneh Hajishirzi1,2 1University of Washington 2Allen Institute for AI {alrope,sewon,lsz,hannaneh}@cs.washington.edu beltagy@allenai.org Abstract Although large language models can be prompted for both zero- and few-shot learn- ing, performance drops significantly when no demonstrations are available. In this pa- per, we introduce Z-ICL, a new zero-shot method that closes the gap by constructing pseudo-demonstrations for a given test input using a raw text corpus. Concretely, pseudo- demonstrations are constructed by (1) finding the nearest neighbors to the test input from the corpus and pairing them with random task la- bels, and (2) applying a set of techniques to reduce the amount of direct copying the model does from the resulting demonstrations. Evalu- ation on nine classification datasets shows that Z-ICL outperforms previous zero-shot methods by a significant margin, and is on par with in- context learning with few-shot labeled training data. Overall, Z-ICL provides a significantly higher estimate of the zero-shot performance levels of a model, and supports future efforts to develop better pseudo-demonstrations that further improve zero-shot results.1 1 Introduction Large language models (LMs) can perform new tasks simply by conditioning on input-label pairs from the training data, known as demonstra- tions (Brown et al., 2020). This in-context learning (ICL) is significantly better than zero-shot methods that do not use demonstrations. Recent work sug- gests that in-context-learning demonstrations are primarily specifying the domain and the format that the target task, instead of providing explicit training signal (Reynolds and McDonell, 2021; Xie et al., 2022; Razeghi et al., 2022; Min et al., 2022). This implies that current zero-shot performance (with no demonstrations) levels must be significantly under- estimated, since all the required information must already be in the model. 1Code available at github.com/alrope123/z-icl. Figure 1: An illustration of Z-ICL with k = 3, making a prediction between great and terrible. Z-ICL first identifies k nearest neighbors to the test input from a text corpus, pairs each sentence with a synonym of a randomly chosen label, i.e., good and bad, and uses in-context learning. In this paper, we introduce Z-ICL: Zero- shot In-Context Learning through creating pseudo- demonstrations, which achieves results on par with in-context learning from gold demonstrations (Fig- ure 1). The key idea is to construct the pseudo- demonstrations following two criteria: (a) they should inform the correct input distribution and the label space, as the k-shot demonstrations do (Xie et al., 2020; Min et al., 2022);2 and (b) they should be constructed to avoid the copying effect—our new observation that the LM predictions are heav- ily influenced by demonstration inputs that are very close to the test input. To satisfy (a), Z-ICL retrieves a set of nearest neighbors from a raw text corpus and assigns a random label to each. To satisfy (b), we propose two techniques. We take physical …

Content truncated to 500 words