Unsupervised Parsing via Constituency Tests

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 522–534, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 522 Self-Supervised Meta-Learning for Few-Shot Natural Language Classification Tasks Trapit Bansal ♦∗and Rishikesh Jha† and Tsendsuren Munkhdalai‡ and Andrew McCallum♦ ♦University of Massachusetts, Amherst †Code for Science and Society ‡Microsoft Research, Montr´eal, Canada Abstract Self-supervised pre-training of transformer models has revolutionized NLP applications. Such pre-training with language modeling ob- jectives provides a useful initial point for pa- rameters that generalize well to new tasks with fine-tuning. However, fine-tuning is still data inefficient — when there are few labeled ex- amples, accuracy can be low. Data efficiency can be improved by optimizing pre-training di- rectly for future fine-tuning with few exam- ples; this can be treated as a meta-learning problem. However, standard meta-learning techniques require many training tasks in or- der to generalize; unfortunately, finding a di- verse set of such supervised tasks is usually dif- ficult. This paper proposes a self-supervised approach to generate a large, rich, meta- learning task distribution from unlabeled text. This is achieved using a cloze-style objective, but creating separate multi-class classification tasks by gathering tokens-to-be blanked from among only a handful of vocabulary terms. This yields as many unique meta-training tasks as the number of subsets of vocabulary terms. We meta-train a transformer model on this dis- tribution of tasks using a recent meta-learning framework. On 17 NLP tasks, we show that this meta-training leads to better few-shot gen- eralization than language-model pre-training followed by finetuning. Furthermore, we show how the self-supervised tasks can be combined with supervised tasks for meta-learning, pro- viding substantial accuracy gains over previ- ous supervised meta-learning. 1 Introduction Self-supervised learning has emerged as an impor- tant training paradigm for learning model parame- ters which are more generalizable and yield better representations for many down-stream tasks. This typically involves learning through labels that come ∗Correspondence: tbansal@cs.umass.edu naturally with data, for example words in natural language. Self-supervised tasks typically pose a su- pervised learning problem that can benefit from lots of naturally available data and enable pre-training of model parameters that act as a useful prior for supervised fine-tuning (Erhan et al., 2010). Masked language modeling (Devlin et al., 2018), and other related approaches (Peters et al., 2018; Howard and Ruder, 2018; Radford et al., 2019), is an example of such a self-supervised task that is behind the success of transformer models like BERT. While self-supervised pre-training is beneficial, it has been recently noted that it is not data-efficient and typically requires large amounts of fine-tuning data for good performance on a target task (Yo- gatama et al., 2019; Bansal et al., 2019). This can be evaluated as a few-shot learning problem, where a model is given only few examples of a new task and is expected to perform well on that task. This paper focuses on this problem of few-shot learn- ing and develops models which demonstrate better few-shot generalization to new tasks. Large scale pre-training suffers from a …

Content truncated to 500 words