A Practical Survey on Zero-Shot Prompt Design for In-Context Learning

Proceedings of Recent Advances in Natural Language Processing, pages 641–647 Varna, Sep 4–6, 2023 https://doi.org/10.26615/978-954-452-092-2_069 641 A Practical Survey on Zero-shot Prompt Design for In-context Learning Yinheng Li Columbia University / New York City li.yinheng@columbia.edu Abstract The remarkable advancements in large lan- guage models (LLMs) have brought about sig- nificant improvements in Natural Language Processing(NLP) tasks. This paper presents a comprehensive review of in-context learn- ing techniques, focusing on different types of prompts, including discrete, continuous, few- shot, and zero-shot, and their impact on LLM performance. We explore various approaches to prompt design, such as manual design, opti- mization algorithms, and evaluation methods, to optimize LLM performance across diverse tasks. Our review covers key research studies in prompt engineering, discussing their method- ologies and contributions to the field. We also delve into the challenges faced in evaluating prompt performance, given the absence of a single ”best” prompt and the importance of considering multiple metrics. In conclusion, the paper highlights the critical role of prompt design in harnessing the full potential of LLMs and provides insights into the combination of manual design, optimization techniques, and rigorous evaluation for more effective and effi- cient use of LLMs in various NLP tasks. 1 Introduction In recent years, transformer-based language models (such as (Raffel et al., 2019), (Lewis et al., 2019), (Brown et al., 2020), (Devlin et al., 2018)) have emerged as a transformative force in the field of artificial intelligence, revolutionizing Natural Lan- guage Understanding(NLU) and Generation(NLG). As model size and training data have evolved, the GPT series has exhibited extraordinary capabilities in a wide range of natural language tasks by rely- ing on a paradigm known as in-context learning. According to (Brown et al., 2020), in-context learn- ing harnesses the context provided by input data to generate appropriate responses or predictions, con- trasting with traditional methods that necessitate explicit task-specific training and fine-tuning on labeled datasets. In-context learning enables large language models to capitalize on vast amounts of data and adapt to various tasks in a flexible and dynamic manner. There are several categories of in-context learning, including zero-shot, one-shot, and few-shot learning. In all types of in-context learning, the key to success lies in effective prompt design, which is occasionally referred to as an ”art.” This survey paper aims to categorize each type of in-context learning, discuss the core principles, ex- amine state-of-the-art design techniques, and ex- plore recent advancements in in-context learning, with a particular focus on zero-shot discrete in- context learning. 2 Definition Although there is no formal definition for prompt design optimization, we follow the principle from (Brown et al., 2020) and provide the definition in (1) for prompt design in in-context learning: P ⋆ = arg max P Exi,yi∈D[S(fθ(P, xi), yi)] (1) Here, xi represents input sentences and features, while yi denotes the target labels. θ signifies the pa- rameters for any Large Language Models (LLMs) or Pretrained Language Models (PLMs), which re- main frozen in the case of in-context learning. fθ represents the output from LLMs …

Content truncated to 500 words