Learning from Task Descriptions

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 1361–1375, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 1361 Learning from Task Descriptions Orion Weller*1, Nicholas Lourie2, Matt Gardner2, Matthew E. Peters2 1Brigham Young University 2Allen Institute for Artificial Intelligence orionw@byu.edu,{nicholasl,mattg,matthewp}@allenai.org Abstract Typically, machine learning systems solve new tasks by training on thousands of examples. In contrast, humans can solve new tasks by read- ing some instructions, with perhaps an exam- ple or two. To take a step toward closing this gap, we introduce a framework for developing NLP systems that solve new tasks after read- ing their descriptions, synthesizing prior work in this area. We instantiate this framework with a new English language dataset, ZEST, structured for task-oriented evaluation on un- seen tasks. Formulating task descriptions as questions, we ensure each is general enough to apply to many possible inputs, thus compre- hensively evaluating a model’s ability to solve each task. Moreover, the dataset’s structure tests specific types of systematic generaliza- tion. We find that the state-of-the-art T5 model achieves a score of 12% on ZEST, leaving a significant challenge for NLP researchers.1 1 Introduction The dominant paradigm in supervised NLP today is learning from examples, where machine learn- ing algorithms are trained using a large set of task- specific input-output pairs. In contrast, humans learn to perform the same task by reading a de- scription, after which they are able to perform the task in a zero-shot manner—indeed, this is how crowd-sourced NLP datasets are constructed. In this paper, we argue that learning from task de- scriptions in this way is a necessary attribute of a general purpose NLP system, and we propose it as a new paradigm to train and test NLP systems. Recent work in NLP has shown significant progress in learning tasks from examples. Large pretrained language models have dramatically im- proved performance on standard benchmarks (Pe- ters et al., 2018; Devlin et al., 2019; Raffel et al., ∗Work done while at the Allen Institute for AI. 1Data, evaluation code, baseline models, and leaderboard at https://allenai.org/data/zest X Y N X Y N M dT (a) (b) T T Figure 1: Comparison of (a) supervised learning from examples with observed input X, output Y , corre- sponding to an unobserved task τ (b) our proposed method of learning from task descriptions where sys- tems can make inferences about unseen tasks τ given a natural language description dτ. 2019) and have shown promising results in zero shot prediction by leveraging their language un- derstanding capabilities (Levy et al., 2017; Zhou et al., 2018; Yin et al., 2019). Despite this progress, there are many serious issues that come with learning from examples. There is an almost infinite number of tasks that a person might wish to solve with a general-purpose NLP system. Learning to solve these tasks by reading a description instead of observing a col- lection of examples would solve the problem of having to create training sets for each language task. Such …

Content truncated to 500 words