Learning to Few-Shot Learn Across Diverse Natural Language Classification Tasks

Proceedings of the 28th International Conference on Computational Linguistics, pages 5108–5123 Barcelona, Spain (Online), December 8-13, 2020 5108 Learning to Few-Shot Learn Across Diverse Natural Language Classification Tasks Trapit Bansal∗† and Rishikesh Jha∗‡ and Andrew McCallum† †University of Massachusetts, Amherst ‡Code for Science and Society {tbansal,rishikeshjha,mccallum}@cs.umass.edu Abstract Pre-trained transformer models have shown enormous success in improving performance on sev- eral downstream tasks. However, fine-tuning on a new task still requires large amounts of task- specific labeled data to achieve good performance. We consider this problem of learning to generalize to new tasks with a few examples as a meta-learning problem. While meta-learning has shown tremendous progress in recent years, its application is still limited to simulated prob- lems or problems with limited diversity across tasks. We develop a novel method, LEOPARD, which enables optimization-based meta-learning across tasks with different number of classes, and evaluate different methods on generalization to diverse NLP classification tasks. LEOP- ARD is trained with the state-of-the-art transformer architecture and shows better generalization to tasks not seen at all during training, with as few as 4 examples per label. Across 17 NLP tasks, including diverse domains of entity typing, natural language inference, sentiment analysis, and several other text classification tasks, we show that LEOPARD learns better initial parame- ters for few-shot learning than self-supervised pre-training or multi-task training, outperforming many strong baselines, for example, yielding 14.6% average relative gain in accuracy on unseen tasks with only 4 examples per label. 1 Introduction Learning to learn (Schmidhuber, 1987; Bengio et al., 1992; Thrun and Pratt, 2012) from limited super- vision is an important problem with widespread application in areas where obtaining labeled data for training large models can be difficult or expensive. We consider this problem of learning in k-shots for natural language processing (NLP) tasks, that is, given k labeled examples of a new NLP task learn to efficiently solve the new task. Recently, self-supervised pre-training of transformer models using lan- guage modeling objectives (Devlin et al., 2018; Radford et al., 2019; Yang et al., 2019) has achieved tremendous success in learning general-purpose parameters which are useful for a variety of downstream NLP tasks. While pre-training is beneficial, it is not optimized for fine-tuning with limited supervision and such models still require large amounts of task-specific data for fine-tuning, in order to achieve good performance (Yogatama et al., 2019). On the other hand, meta-learning methods have been proposed as effective solutions for few-shot learning. Existing applications of such meta-learning methods have shown improved performance in few-shot learning for vision tasks such as learning to classify new image classes within a similar dataset. However, these applications are often limited to simulated datasets where each classification label is considered a task. Moreover, their application in NLP has followed a similar trend (Han et al., 2018; Yu et al., 2018; Guo et al., 2018; Mi et al., 2019; Geng et al., 2019). Since the input space of natural language is shared across all NLP tasks, it is possible that a …

Content truncated to 500 words