Reordering Examples Helps during Priming-based Few-Shot Learning

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4507–4518 August 1–6, 2021. ©2021 Association for Computational Linguistics 4507 Reordering Examples Helps during Priming-based Few-Shot Learning Sawan Kumar Indian Institute of Science, Bangalore sawankumar@iisc.ac.in Partha Talukdar Indian Institute of Science, Bangalore ppt@iisc.ac.in Abstract The ability to learn from limited data, or few- shot learning, is a desirable and often critical requirement for NLP systems. While many existing methods do poorly at learning from a handful of examples, large pretrained lan- guage models have recently been shown to be efficient few-shot learners. One approach to few-shot learning, which does not require finetuning of model parameters, is to aug- ment the language model’s input with prim- ing text which is typically constructed using task specific descriptions and examples. In this work, we further explore priming-based few-shot learning, with focus on using exam- ples as prompts. We show that presenting ex- amples in the right order is key for general- ization. We introduce PERO (Prompting with Examples in the Right Order), where we for- mulate few-shot learning as search over the set of permutations of the training examples. We show that PERO can learn to generalize effi- ciently using as few as 10 examples, in con- trast to existing approaches. While the new- line token is a natural choice for separating the examples in the prompt, we show that learn- ing a new separator token can potentially pro- vide further gains in performance. We demon- strate the effectiveness of the proposed method on the tasks of sentiment classification, natural language inference and fact retrieval. Finally, we analyze the learned prompts to reveal novel insights, including the idea that two training examples in the right order alone can provide competitive performance for sentiment classi- fication and natural language inference. 1 Introduction The ability to learn from a few examples, or few- shot learning, as generally understood to be pos- sessed by humans, is a desirable property for Nat- ural Language Processing (NLP) systems as well. It is critical in scenarios where collecting large amounts of data is expensive. It is also important to enable a personalized Artificial Intelligence (AI) experience, where a single user is expected to use an AI agent to perform a task demonstrated through a handful of examples.1 Pretrained language models (Devlin et al., 2019; Liu et al., 2019; Raffel et al., 2020) have recently been shown to be exceedingly good at several benchmark NLP tasks (Wang et al., 2018, 2019). Traditionally the parameters of these language mod- els have been finetuned on task specific datasets to achieve the aforementioned performance gains, often requiring large amounts of data. Brown et al. (2020) show that large pretrained language models (GPT3) are also efficient few-shot learners. Few- shot learning is achieved using task descriptions and labeled examples as prompts. Remarkably, with this priming-based approach and without need- ing any parameter updates, GPT3 often performs comparable to traditional finetuning-based super- vised systems which use much larger datasets. One could argue that the task …

Content truncated to 500 words