CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7163–7189 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 7163 CROSSFIT : A Few-shot Learning Challenge for Cross-task Generalization in NLP Qinyuan Ye Bill Yuchen Lin Xiang Ren University of Southern California {qinyuany, yuchen.lin, xiangren}@usc.edu Abstract Humans can learn a new language task effi- ciently with only few examples, by leveraging their knowledge obtained when learning prior tasks. In this paper, we explore whether and how such cross-task generalization ability can be acquired, and further applied to build bet- ter few-shot learners across diverse NLP tasks. We introduce CROSSFIT , a problem setup for studying cross-task generalization ability, which standardizes seen/unseen task partitions, data access during different learning stages, and the evaluation protocols. To instantiate different seen/unseen task partitions in CROSS- FIT and facilitate in-depth analysis, we present the NLP Few-shot Gym, a repository of 160 diverse few-shot NLP tasks created from open- access NLP datasets and converted to a uni- fied text-to-text format. Our analysis reveals that the few-shot learning ability on unseen tasks can be improved via an upstream learn- ing stage using a set of seen tasks. We also observe that the selection of upstream learning tasks can significantly influence few-shot per- formance on unseen tasks, asking further anal- ysis on task similarity and transferability.1 1 Introduction Pre-trained language models fine-tuned with abun- dant task-specific data have become the predomi- nant recipe for state-of-the-art results in NLP. How- ever, these approaches are heavily dependent on large-scale labeled datasets that are expensive to create, and the resulting models still generalize poorly to out-of-distribution inputs created with small, harmless perturbations (Ribeiro et al., 2020). In retrospect, researchers have advocated for build- ing more human-like, general linguistic intelli- gence that can “reuse previously acquired knowl- edge about a language and adapt to a new task quickly” (Yogatama et al., 2019; Linzen, 2020). 1Our code is at https://github.com/INK-USC/CrossFit. Others (regression, etc.) NLP Few-shot Gym A repository of 160 diverse few-shot tasks in NLP Stage 1. Upstream Learning (Multitask Learning, Meta-learning, etc.) Text-to-text Transformer Text-to-text Transformer Stage 2. Downstream Fine-tuning Training Tasks (e.g., non-cls. tasks) Test Tasks (e.g., classification) Classification Question Asnwering Conditional Generation The Cr ossFit Challenge Can we build a few-shot learner to generalize beyond task boundaires? Figure 1: We present the CROSSFIT Challenge to study cross-task generalization in a diverse task distribution. To support this problem setting, we introduce the NLP Few-shot Gym, a repository of 160 diverse few-shot, text-to-text tasks in NLP. Existing work has approached this problem via better few-shot fine-tuning, by re-formulating tar- get tasks into cloze questions that resembles the pre- training objective (Schick and Schütze, 2020a,b), generating prompts and using demonstrations (Gao et al., 2020). Such progress primarily focus on improving instance-level generalization, i.e., how to better generalize from few labeled instances to make predictions about new instances, within the scope of one individual task. From a broader per- spective, human-like learning ability also benefits from task-level generalization, or cross-task gener- …

Content truncated to 500 words