Instruction Induction: From Few Examples to Natural Language Task Descriptions

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 1935–1952 July 9-14, 2023 ©2023 Association for Computational Linguistics Instruction Induction: From Few Examples to Natural Language Task Descriptions Or Honovichτ Uri Shahamτ Samuel R. Bowmanν Omer Levyτµ τ Tel Aviv University ν New York University µ Meta AI Abstract Large language models are able to perform a task by conditioning on a few input-output demonstrations – a paradigm known as in- context learning. We show that language mod- els can explicitly infer an underlying task from a few demonstrations by prompting them to generate a natural language instruction that fits the examples. To explore this ability, we intro- duce the instruction induction challenge, com- pile a dataset consisting of 24 tasks, and define a novel evaluation metric based on executing the generated instruction. We discover that, to a large extent, the ability to generate instruc- tions does indeed emerge when using a model that is both large enough and aligned to fol- low instructions; InstructGPT achieves 65.7% of human performance in our execution-based metric, while the original GPT-3 model reaches only 9.8% of human performance. This surpris- ing result suggests that instruction induction might be a viable learning paradigm in and of itself, where instead of fitting a set of latent continuous parameters to the data, one searches for the best description in the natural language hypothesis space.1 1 Introduction Large language models (LMs) can perform unseen tasks by conditioning on a few labeled examples, effectively inferring the underlying tasks through a process known as in-context learning (Brown et al., 2020). However, task inference is implicit, and the ability of models to explicitly reason about it remains unexplored. In this work, we show that LMs can explicitly describe an underlying task, in natural language, given a few labeled examples. We introduce the instruction induction challenge, in which a model is provided with a few input- output demonstrations, and is requested to gener- ate a natural language instruction describing the 1Our code and data are publicly available at https://github.com/orhonovich/ instruction-induction connection between the input-output pairs. In our experiments, inducing instructions is done in a zero- shot manner by simply prompting the models to explain a small set of given demonstrations, as shown in Figure 1; we do not perform fine-tuning or use any labeled instruction induction data. We examine instruction induction on 24 tasks, ranging from morphosyntactic tasks to style trans- fer and sentiment analysis. Since our goal is to shed light on the phenomenon of instruction induction, we focus on tasks that have clear and simple instruc- tions. As a basic evaluation protocol, we collect human annotations and use them as gold-standard references; the generated instructions are then com- pared to these references using BERTScore (Zhang et al., 2020). Moreover, we suggest a novel eval- uation metric for instruction induction: execution accuracy. The execution accuracy of a generated instruction is measured by testing whether LMs can correctly perform the task in a …

Content truncated to 500 words