Incremental Few-shot Text Classification with Multi-round New Classes: Formulation, Dataset and System

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1351–1360 June 6–11, 2021. ©2021 Association for Computational Linguistics 1351 Incremental Few-shot Text Classification with Multi-round New Classes: Formulation, Dataset and System Congying Xia1∗, Wenpeng Yin2∗, Yihao Feng3, Philip Yu1 1University of Illinois at Chicago, Chicago, IL, USA 2Salesforce Research, Palo Alto, CA, USA 3University of Texas at Austin, Austin, TX, USA {cxia8, psyu}@uic.edu, wyin@salesforce.com, yihao@cs.utexas.edu Abstract Text classification is usually studied by label- ing natural language texts with relevant cate- gories from a predefined set. In the real world, new classes might keep challenging the ex- isting system with limited labeled data. The system should be intelligent enough to recog- nize upcoming new classes with a few exam- ples. In this work, we define a new task in the NLP domain, incremental few-shot text classi- fication, where the system incrementally han- dles multiple rounds of new classes. For each round, there is a batch of new classes with a few labeled examples per class. Two major challenges exist in this new task: (i) For the learning process, the system should incremen- tally learn new classes round by round with- out re-training on the examples of preceding classes; (ii) For the performance, the system should perform well on new classes without much loss on preceding classes. In addition to formulating the new task, we also release two benchmark datasets 1 in the incremental few- shot setting: intent classification and relation classification. Moreover, we propose two en- tailment approaches, ENTAILMENT and HY- BRID, which show promise for solving this novel problem. 1 Introduction Text classification has achieved great success in the past decades with the development of deep learning techniques (Kowsari et al., 2019; Li et al., 2020). However, decent performance highly relies on the availability of large-scale task-specific train- ing data. Recently, few-shot text classification (Yu et al., 2018; Geng et al., 2019; Xia et al., 2020a) has attracted increasing attention from the NLP community since it is unlikely to have large-scale labeled data for new classes in the real world. ∗Indicates Equal Contribution. 1Code & Data are available at https://github.com/ congyingxia/IncrementalFSTC. Typically, few-shot text classification is formu- lated like this: the system first sees a set of base classes Cb that have a large number of labeled ex- amples, then a group of new classes Cn is provided with k examples per class. For a testing instance, the system is required to search for its label in the space of Cb ∪ Cn or merely Cn. However, this setting might not suitable for real scenarios. First, base classes with rich annotations might not be available at the beginning. It happens whenever you want to build a system from scratch. Second, take the bank’s customer service system as an ex- ample, queries with new intents are continuously appearing (e.g., by a sequence of rounds) without enough labeled data. The system should be able to keep learning and recognizing new intents …

Content truncated to 500 words