Learning To Retrieve Prompts for In-Context Learning

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2655 - 2671 July 10-15, 2022 ©2022 Association for Computational Linguistics Learning To Retrieve Prompts for In-Context Learning Ohad Rubin Jonathan Herzig Jonathan Berant The Blavatnik School of Computer Science, Tel Aviv University {ohad.rubin,jonathan.herzig,joberant}@cs.tau.ac.il Abstract In-context learning is a recent paradigm in nat- ural language understanding, where a large pre- trained language model (LM) observes a test in- stance and a few training examples as its input, and directly decodes the output without any up- date to its parameters. However, performance has been shown to strongly depend on the se- lected training examples (termed prompts). In this work, we propose an efficient method for retrieving prompts for in-context learning us- ing annotated data and an LM. Given an input- output pair, we estimate the probability of the output given the input and a candidate train- ing example as the prompt, and label training examples as positive or negative based on this probability. We then train an efficient dense retriever from this data, which is used to re- trieve training examples as prompts at test time. We evaluate our approach on three sequence-to- sequence tasks where language utterances are mapped to meaning representations, and find that it substantially outperforms prior work and multiple baselines across the board. 1 Introduction The striking language skills and world knowledge embedded in large pre-trained language models (LMs) (Devlin et al., 2019; Petroni et al., 2019; Raf- fel et al., 2020; Brown et al., 2020) have recently led to in-context learning, a new paradigm in natu- ral language understanding. Under this paradigm, a language model is given a prompt, which typi- cally contains a few training examples, as well as a test instance as input, and generates the output for the test instance directly, without any update to its parameters. This approach was first introduced in GPT-3 (Brown et al., 2020), but has quickly spread to other LMs (Lieber et al., 2021; Du et al., 2021; Rae et al., 2021). An attractive property of in-context learning is that it provides a single model for multiple lan- guage understanding tasks. However, Liu et al. Retriever Retriever Index What is the longest river in the smallest state in the usa? 1) states 2) size of #1 3) #1 where #2 is the lowest 4) rivers of #3 5) how long are #4 6) #4 where #5 is the highest Which states border the shortest river in the usa? 1) the usa 2) rivers of #1 3) how long are #2 4) #2 where #3 is the lowest 5) border states of #4 Which states border the longest river in the usa? 1) the usa 2) rivers of #1 3) how long are #2 4) #2 where #3 is the highest 5) border states of #4 1) rivers 2) #1 in the usa 3) lengths of #2 4) #2 where #3 is longest 5) length of #4 Inference LM …

Content truncated to 500 words