Continual Few-Shot Learning for Text Classification

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5688–5702 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 5688 Continual Few-Shot Learning for Text Classification Ramakanth Pasunuru1,2 Veselin Stoyanov2 Mohit Bansal1 1UNC Chapel Hill 2Facebook AI {ram,mbansal}@cs.unc.edu, ves@fb.com Abstract Natural Language Processing (NLP) is increas- ingly relying on general end-to-end systems that need to handle many different linguistic phenomena and nuances. For example, a Nat- ural Language Inference (NLI) system has to recognize sentiment, handle numbers, perform coreference, etc. Our solutions to complex problems are still far from perfect, so it is im- portant to create systems that can learn to cor- rect mistakes quickly, incrementally, and with little training data. In this work, we propose a continual few-shot learning (CFL) task, in which a system is challenged with a difficult phenomenon and asked to learn to correct mis- takes with only a few (10 to 15) training ex- amples. To this end, we first create bench- marks based on previously annotated data: two NLI (ANLI and SNLI) and one sentiment anal- ysis (IMDB) datasets. Next, we present var- ious baselines from diverse paradigms (e.g., memory-aware synapses and Prototypical net- works) and compare them on few-shot learn- ing and continual few-shot learning setups. Our contributions are in creating a benchmark suite1 and evaluation protocol for continual few-shot learning on the text classification tasks, and making several interesting observa- tions on the behavior of similarity-based meth- ods. We hope that our work serves as a useful starting point for future work on this important topic. 1 Introduction Large end-to-end neural models are becoming more pervasive in Computer Vision (CV) and Nat- ural Language Processing (NLP). In NLP in par- ticular, large language models such as BERT (De- vlin et al., 2019) fine-tuned end-to-end for a task, have advanced the state-of-the-art for many prob- lems such as classification, Natural Language Infer- ence (NLI), and Question Answering (QA) (Devlin 1https://github.com/ ramakanth-pasunuru/CFL-Benchmark et al., 2019; Liu et al., 2019; Wang et al., 2019). End-to-end models are conceptually simpler than the previously-popular pipelined models, making them easier to deploy and maintain. However, be- cause large end-to-end models are black-boxes, it is difficult to correct the mistakes that they make. Practical, real-world applications of NLP require such mistakes to be corrected on the fly as the system operates. For example, when a translation system makes a harmful mistake (e.g., translates “EMNLP” to “ICML”), a phrase-based system can be corrected by finding and modifying the respon- sible entries in the phrase table (Zens et al., 2002), whereas there is no equivalent way to correct that in an end-to-end neural MT system. Similarly, sys- tems have been shown to exhibit bias (e.g., gender or racial stereotypes) toward certain inputs of text, which we want to correct via few examples on the fly. Further, the examples that provide supervision to correct mistakes or learn a phenomenon are often hard or impossible to acquire (e.g., due to privacy or ethics issues) (Wang et al., 2020). Hence, …

Content truncated to 500 words