Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5085 - 5109 December 7-11, 2022 ©2022 Association for Computational Linguistics SUPER-NATURALINSTRUCTIONS: Generalization via Declarative Instructions on 1600+ NLP Tasks ♢Yizhong Wang2 ♢Swaroop Mishra3 ♣Pegah Alipoormolabashi4 ♣Yeganeh Kordi5 Amirreza Mirzaei4 Anjana Arunkumar3 Arjun Ashok6 Arut Selvan Dhanasekaran3 Atharva Naik7 David Stap8 Eshaan Pathak9 Giannis Karamanolakis10 Haizhi Gary Lai11 Ishan Purohit12 Ishani Mondal13 Jacob Anderson3 Kirby Kuznia3 Krima Doshi3 Maitreya Patel3 Kuntal Kumar Pal3 Mehrad Moradshahi14 Mihir Parmar3 Mirali Purohit15 Neeraj Varshney3 Phani Rohitha Kaza3 Pulkit Verma3 Ravsehaj Singh Puri3 Rushang Karia3 Shailaja Keyur Sampat3 Savan Doshi3 Siddhartha Mishra16 Sujan Reddy17 Sumanta Patro18 Tanay Dixit19 Xudong Shen20 Chitta Baral3 Yejin Choi1,2 Noah A. Smith1,2 Hannaneh Hajishirzi1,2 Daniel Khashabi21 1Allen Institute for AI 2Univ. of Washington 3Arizona State Univ. 4Sharif Univ. of Tech. 5Tehran Polytechnic 6PSG College of Tech. 7IIT Kharagpur 8Univ. of Amsterdam 9UC Berkeley 10Columbia Univ. 11Factored AI 12Govt. Polytechnic Rajkot 13Microsoft Research 14Stanford Univ. 15Zycus Infotech 16Univ. of Massachusetts Amherst 17National Inst. of Tech. Karnataka 18TCS Research 19IIT Madras 20National Univ. of Singapore 21Johns Hopkins Univ. Abstract How well can NLP models generalize to a va- riety of unseen tasks when provided with task instructions? To address this question, we first introduce SUPER-NATURALINSTRUCTIONS,1 a benchmark of 1,616 diverse NLP tasks and their expert-written instructions. Our collec- tion covers 76 distinct task types, including but not limited to classification, extraction, in- filling, sequence tagging, text rewriting, and text composition. This large and diverse collec- tion of tasks enables rigorous benchmarking of cross-task generalization under instructions— training models to follow instructions on a sub- set of tasks and evaluating them on the remain- ing unseen ones. Furthermore, we build Tk-INSTRUCT, a trans- former model trained to follow a variety of in-context instructions (plain language task def- initions or k-shot examples). Our experiments show that Tk-INSTRUCT outperforms existing instruction-following models such as Instruct- GPT by over 9% on our benchmark despite being an order of magnitude smaller. We fur- ther analyze generalization as a function of var- ious scaling parameters, such as the number of observed tasks, the number of instances per task, and model sizes. We hope our dataset and model facilitate future progress towards more general-purpose NLP models.2 1 Introduction The NLP community has witnessed great progress in building models for generalization to unseen tasks via in-context instructions (Mishra et al., 1SUPER-NATURALINSTRUCTIONS represents a super- sized expansion of NATURALINSTRUCTIONS (Mishra et al., 2022b) which had 61 tasks. 2The dataset, models, and a leaderboard can be found at https://instructions.apps.allenai.org. ♢ Co-first authors ♣ Co-second authors • Input: “Context: … ‘That's fantastic, I'm glad we came to something we both agree with.’ Utterance: ‘Me too. I hope you have a wonderful camping trip.’” • Output: “Yes” • Explanation: “The participant engages in small talk when wishing their opponent to have a wonderful trip.” • Input: “Context: … ‘Sounds good, I need food the most, what is your most needed item?!’ Utterance: ‘My item is food too’.” • Output: “Yes” …

Content truncated to 500 words