Diverse Distributions of Self-Supervised Tasks for Meta-Learning in NLP

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5812–5824 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 5812 Diverse Distributions of Self-Supervised Tasks for Meta-Learning in NLP Trapit Bansal♠∗ † and Karthick Gunasekaran♠ and Tong Wang♦ and Tsendsuren Munkhdalai♣† and Andrew McCallum♠ ♠ University of Massachusetts Amherst ♦ Microsoft Research, Montréal, Canada ♣ Google Research Abstract Meta-learning considers the problem of learn- ing an efficient learning process that can lever- age its past experience to accurately solve new tasks. However, the efficacy of meta-learning crucially depends on the distribution of tasks available for training, and this is often assumed to be known a priori or constructed from lim- ited supervised datasets. In this work, we aim to provide task distributions for meta-learning by considering self-supervised tasks automati- cally proposed from unlabeled text, to enable large-scale meta-learning in NLP. We design multiple distributions of self-supervised tasks by considering important aspects of task diver- sity, difficulty, type, domain, and curriculum, and investigate how they affect meta-learning performance. Our analysis shows that all these factors meaningfully alter the task distribu- tion, some inducing significant improvements in downstream few-shot accuracy of the meta- learned models. Empirically, results on 20 downstream tasks show significant improve- ments in few-shot learning – adding up to +4.2% absolute accuracy (on average) to the previous unsupervised meta-learning method, and perform comparably to supervised meth- ods on the FewRel 2.0 benchmark. 1 Introduction Humans show a remarkable capability to accu- rately solve a wide range of problems efficiently – utilizing a limited amount of computation and experience. Deep learning models, by stark con- trast, can be trained to be highly accurate on a nar- row task while being highly inefficient in terms of the amount of compute and data required to reach that accuracy. Within natural language processing (NLP), recent breakthroughs in unsupervised pre- training have enabled reusable models that can be applied to many NLP tasks, however, learning of ∗Correspondence: tbansal@cs.umass.edu. † Part of the work was done at Microsoft Research. new tasks is still inefficient (Yogatama et al., 2019; Bansal et al., 2020a; Linzen, 2020). Meta-learning (Schmidhuber, 1987; Bengio et al., 1992; Thrun and Pratt, 2012) treats the learning process itself as a learning problem from data, with the goal of learning systems that can generalize to new tasks efficiently. This has the potential to produce few- shot learners that can accurately solve a wide range of new tasks. However, meta-learning requires a distribution over tasks with relevant labeled data that can be difficult to obtain, severely limiting the practical utility of meta-learning methods. In the supervised setting, in particular, meta- learning task distribution is often defined by sub- sampling from the classes in a classification prob- lem over a fixed dataset (Vinyals et al., 2016). This not only limits the applicability of meta-learning to the underlying classification problem, but also requires a diverse set of supervised datasets with a large number of classes to enable learning. Self- supervised meta-learning, on the other …

Content truncated to 500 words