Open World Classification with Adaptive Negative Samples

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4378 - 4392 December 7-11, 2022 ©2022 Association for Computational Linguistics Open World Classification with Adaptive Negative Samples Ke Bai1, Guoyin Wang2, Jiwei Li3, Sunghyun Park2, Sungjin Lee2, Puyang Xu2, Ricardo Henao1, Lawrence Carin4 1Duke University 2Amazon 3Zhejiang University 4KAUST {ke.bai, ricardo.henao}@duke.edu, {guoyiwan, sunghyu, sungjinl, puyax}@amazon.com, jiwei_li@zju.edu.cn, larry.carin@kaust.edu.sa Abstract Open world classification is a task in natural language processing with key practical rele- vance and impact. Since the open or unknown category data only manifests in the inference phase, finding a model with a suitable deci- sion boundary accommodating for the identi- fication of known classes and discrimination of the open category is challenging. The per- formance of existing models is limited by the lack of effective open category data during the training stage or the lack of a good mecha- nism to learn appropriate decision boundaries. We propose an approach based on adaptive negative samples (ANS) designed to generate effective synthetic open category samples in the training stage and without requiring any prior knowledge or external datasets. Empirically, we find a significant advantage in using auxil- iary one-versus-rest binary classifiers, which effectively utilize the generated negative sam- ples and avoid the complex threshold-seeking stage in previous works. Extensive experiments on three benchmark datasets show that ANS achieves significant improvements over state- of-the-art methods. 1 Introduction Standard supervised classification assumes that all categories expected in the testing phase have been fully observed while training, i.e., every sample is assigned to one known category as illustrated in Figure 1(a). This may not always be satisfied in practical applications, such as dialogue intention classification, where new intents are expected to emerge. Consequently, it is desirable to have a classifier capable of discriminating whether a given sample belongs to a known or an unknown category, e.g., the red samples in Figure 1(a). This problem can be understood as a (C + 1) classification prob- lem, where C is the number of known categories and the additional category is reserved for open (unknown) samples. This scenario is also known C1 C0 C2 C0 (a) Multi-class Supervised (b) Optimal Boundary (c) Adaptive Decision Boundary (d) Ours Correctly/Wrongly Classified Open Category Samples Distribution of Known Categories C1 C0 C2 C0 C1 C0 C2 C0 C1 C0 C2 C0 Synthesized Negative Samples Decision Boundary Figure 1: Illustration of previous methods and our pro- posed one. C0, C1, and C2 are three categories. The boundary is used to discriminate known and open (un- known) categories. (a) Boundary learned by supervised learning. (b) Optimal decision boundary. (c) ADB method which has a closed decision boundary, but may capture irrelevant points. (d) Proposed ANS method. as multi-class open set recognition (Scheirer et al., 2014). To discriminate the known from the open sam- ples during inference, it is necessary to create a clear classification boundary that separates the known from the open category. However, the lack of open category samples during training makes this problem challenging. Current research …

Content truncated to 500 words