Reframing Instructional Prompts to GPTk’s Language

Findings of the Association for Computational Linguistics: ACL 2022, pages 589 - 612 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Reframing Instructional Prompts to GPTk’s Language Swaroop Mishra3 Daniel Khashabi1 Chitta Baral3 Yejin Choi1,2 Hannaneh Hajishirzi1,2 1Allen Institute for AI 2University of Washington 3Arizona State University Abstract What kinds of instructional prompts are easier to follow for Language Models (LMs)? We study this question by conducting extensive empirical analysis that shed light on important features of successful instructional prompts. Specifically, we study several classes of re- framing techniques for manual reformulation of prompts into more effective ones. Some examples include decomposing a complex task instruction into multiple simpler tasks or item- izing instructions into sequential steps. Our experiments compare the zero-shot and few- shot performance of LMs prompted with re- framed instructions on 12 NLP tasks across 6 categories. Compared with original instruc- tions, our reframed instructions lead to signif- icant improvements across LMs with differ- ent sizes. For example, the same reframed prompts boost few-shot performance of GPT3- series and GPT2-series by 12.5% and 6.7% respectively averaged over all tasks. Further- more, reframed instructions reduce the num- ber of examples required to prompt LMs in the few-shot setting. We hope these empirically- driven techniques will pave the way towards more effective future prompting algorithms. 1 Introduction Prompting language models (LMs) (Liu et al., 2021a) has made NLP modules accessible to non- expert users through plain text instructions1 of NLP tasks. Such task instructions written by non-expert users are often long and contain abstract descrip- tions which are not easy to follow for LMs, as evident by their low performance (Efrat and Levy, 2020; Mishra et al., 2022). However, it is not quite clear whether this is due to the inherent difficulty of the target tasks or an artifact of the complex phrasing of their language instructions. 1We focus on instructional prompts (Efrat and Levy, 2020) as opposed to exemplar prompts which are already well- studied (Brown et al., 2020; Lu et al., 2021). You are given passages that contain mentions of names of people, places, or things. Your job is to write questions that evaluate one's understanding of pronouns (she, her, him, his, their, etc.) or other mentions to people, places, or things to which they may refer. Raw Task Definition Generate names of persons, places or things from the passage. Generate a question from the passage with name as the answer. Based on the passage, generate a question that contains the name. Generate a question using $Q1 and $Q2 with $A1 as the answer Biden Q2: Who is the president of US? A2: Biden Q1: What is Biden's birthplace? A1: Scranton What is the birthplace of the person who is the president of US? Reframed Task Definition Reframing Figure 1: GPT3 has difficulty in writing questions that require entity coreference resolutions based on a single lengthy prompt (top, in yellow), however, it succeeds in solving a manually reframed task that has four sim- pler sub-steps (bottom, in …

Content truncated to 500 words