DReCa: A General Task Augmentation Strategy for Few-Shot Natural Language Inference

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1113–1125 June 6–11, 2021. ©2021 Association for Computational Linguistics 1113 DReCa: A General Task Augmentation Strategy for Few-Shot Natural Language Inference Shikhar Murty Tatsunori B. Hashimoto Christopher D. Manning Computer Science Department, Stanford University {smurty,thashim,manning}@cs.stanford.edu Abstract Meta-learning promises few-shot learners that quickly adapt to new distributions by repurpos- ing knowledge acquired from previous train- ing. However, we believe meta-learning has not yet succeeded in NLP due to the lack of a well-defined task distribution, leading to attempts that treat datasets as tasks. Such an ad hoc task distribution causes problems of quantity and quality. Since there’s only a handful of datasets for any NLP prob- lem, meta-learners tend to overfit their adap- tation mechanism and, since NLP datasets are highly heterogeneous, many learning episodes have poor transfer between their support and query sets, which discourages the meta-learner from adapting. To alleviate these issues, we propose DRECA (Decomposing datasets into Reasoning Categories), a simple method for discovering and using latent reasoning categories in a dataset, to form additional high quality tasks. DRECA works by split- ting examples into label groups, embedding them with a finetuned BERT model and then clustering each group into reasoning cate- gories. Across four few-shot NLI problems, we demonstrate that using DRECA improves the accuracy of meta-learners by 1.5–4%. 1 Introduction A key desideratum for human-like understanding is few-shot adaptation. Adaptation is central to many NLP applications since new concepts and words appear often, leading to distribution shifts. People can effortlessly deal with these distribution shifts by learning these new concepts quickly and we would like our models to have similar capabilities. While finetuning large pre-trained transformers is one way to facilitate this adaptation, this procedure requires thousands of samples where humans might require only a few. Can these pre-trained transformers be made to achieve few-shot adaptation? One promising di- rection is meta-learning (Schmidhuber, 1987; Ben- Figure 1: Overview of our approach. We embed all ex- amples with BERT, and then cluster within each label group separately (red and green correspond to entail- ment and not_entailment respectively). Then, we group clusters from distinct label groups to form tasks. gio et al., 1997). Meta-learning promises few-shot classifiers that can adapt to new tasks by repur- posing skills acquired from training tasks. An im- portant prerequisite for successful application of meta-learning is a task-distribution from which a large number of tasks can be sampled to train the meta-learner. While meta-learning is very appeal- ing, applications in NLP have thus far proven chal- lenging due to the absence of a well-defined set of tasks that correspond to re-usable skills. This has led to less effective ad hoc alternatives, like treating entire datasets as tasks. Treating entire datasets as tasks has two major issues. The first issue is learner overfitting (Rajen- dran et al., 2020), where a meta-learner overfits its adaptation mechanism to the small number of training tasks, since there’s …

Content truncated to 500 words