Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 8086 - 8098 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity Yao Lu† Max Bartolo† Alastair Moore‡ Sebastian Riedel† Pontus Stenetorp† †University College London ‡Mishcon de Reya LLP {yao.lu,m.bartolo,s.riedel,p.stenetorp}@cs.ucl.ac.uk alastair.moore@mishcon.com Abstract When primed with only a handful of training samples, very large, pretrained language mod- els such as GPT-3 have shown competitive re- sults when compared to fully-supervised, fine- tuned, large, pretrained language models. We demonstrate that the order in which the sam- ples are provided can make the difference be- tween near state-of-the-art and random guess performance: essentially some permutations are “fantastic” and some not. We analyse this phenomenon in detail, establishing that: it is present across model sizes (even for the largest current models), it is not related to a specific subset of samples, and that a given good per- mutation for one model is not transferable to another. While one could use a development set to determine which permutations are per- formant, this would deviate from the true few- shot setting as it requires additional annotated data. Instead, we use the generative nature of language models to construct an artificial development set and based on entropy statis- tics of the candidate permutations on this set, we identify performant prompts. Our method yields a 13% relative improvement for GPT- family models across eleven different estab- lished text classification tasks. 1 Introduction Large pretrained language models (PLMs, De- vlin et al., 2019; Peters et al., 2018; Raffel et al., 2020; Liu et al., 2019; Yang et al., 2019; Radford et al., 2019) have shown remarkable performance when conditioned with an appropriate textual con- text (Petroni et al., 2019, 2020; Jiang et al., 2020; Shin et al., 2020; Davison et al., 2019). For exam- ple, when conditioned on a long document and a “TL;DR:” token, they can generate a summary of said document, and when provided a partial ques- tion (“The theory of relativity was developed by __”), they can generate the correct answer. Perhaps most strikingly, when primed with a context con- sisting of very few training examples, they produce 0.1 0.3 0.8 1.5 2.7 6.7 13 175 Model Parameters (Billion) 50 60 70 80 90 100 SST-2 Accuracy (%) 0.1 0.3 0.8 1.5 2.7 6.7 13 175 Model Parameters (Billion) 50 60 70 80 90 100 Subj Accuracy (%) Figure 1: Four-shot performance for 24 different sam- ple orders across different sizes of GPT-family models (GPT-2 and GPT-3) for the SST-2 and Subj datasets. text classification results that can match those of fully supervised models. This type of few shot set- ting, is commonly referred to as “In-context Learn- ing” (Brown et al., 2020). A core component of in-context learning is the text-based prompt that serves as the context. Com- posing a prompt requires: (i) text linearisation us- ing a template; and (ii) training sample concate- nation (See …

Content truncated to 500 words