Cross-Task Generalization via Natural Language Crowdsourcing Instructions

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 3470 - 3487 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Cross-Task Generalization via Natural Language Crowdsourcing Instructions Swaroop Mishra3˚ Daniel Khashabi1 Chitta Baral3 Hannaneh Hajishirzi1,2 1Allen Institute for AI 2University of Washington 3Arizona State University Abstract Humans (e.g., crowdworkers) have a remark- able ability in solving different tasks, by sim- ply reading textual instructions that define them and looking at a few examples. Despite the success of the conventional supervised learning on individual datasets, such mod- els often struggle with generalization across tasks (e.g., a question-answering system can- not solve classification tasks). A long-standing challenge in AI is to build a model that learns a new task by understanding the human- readable instructions that define it. To study this, we introduce NATURAL INSTRUCTIONS, a dataset of 61 distinct tasks, their human- authored instructions, and 193k task instances (input-output pairs). The instructions are ob- tained from crowdsourcing instructions used to create existing NLP datasets and mapped to a unified schema. Using this meta-dataset, we measure cross-task generalization by train- ing models on seen tasks and measuring gen- eralization to the remaining unseen ones. We adopt generative pre-trained language models to encode task-specific instructions along with input and generate task output. Our results indicate that models benefit from instructions when evaluated in terms of generalization to unseen tasks (19% better for models utilizing instructions). These models, however, are far behind an estimated performance upperbound, indicating significant room for more progress in this direction.1 1 Introduction We have witnessed great progress in solving many NLP datasets through fine-tuning pre-trained lan- guage models (LMs) (Peters et al., 2018; Brown et al., 2020). More recent studies show tremendous promise in generalization within the set of observed tasks through multi-task training and unified en- coding (Khashabi et al., 2020; Aghajanyan et al., ˚Work done while interning at Allen Institute for AI. 1Dataset is available at https://instructions. apps.allenai.org grammar check tagging essential phrases question typing answering questions Input: She chose to make a salad for lunch on Sunday. Question: how long did it take for her to make a salad? Crowdsourcing Instruction: List all the words that are essential for answering it correctly. [...] Crowdsourcing Instruction: Label the type of the temporal phenomena in the question. Example are [...] Output: 30mins Output: making salad Output: no ? supervision with seen tasks Output: Event duration ? evaluation on unseen tasks Crowdsourcing Instruction: Label "yes" if the sentence contains any grammatical issues. Otherwise, [...] Crowdsourcing Instruction: Answer the provided question based on a given [...] Figure 1: We construct the NATURAL INSTRUCTIONS dataset from crowdsourcing instructions and instances of different NLP datasets. We study if models can learn from seen tasks and generalize to unseen tasks given their natural crowdsourcing instructions. 2021). However, cross-task generalization – gener- alization to unseen tasks – has generally remained under-explored. For example, can we supervise a model with instances of grammar checking or question answering …

Content truncated to 500 words