Breaking the Closed World Assumption in Text Classification

Proceedings of NAACL-HLT 2016, pages 506–514, San Diego, California, June 12-17, 2016. c⃝2016 Association for Computational Linguistics Breaking the Closed World Assumption in Text Classification Geli Fei and Bing Liu Department of Computer Science University of Illinois at Chicago gfei2@uic.edu, liub@cs.uic.edu Abstract Existing research on multiclass text classifica- tion mostly makes the closed world assump- tion, which focuses on designing accurate classifiers under the assumption that all test classes are known at training time. A more re- alistic scenario is to expect unseen classes during testing (open world). In this case, the goal is to design a learning system that classi- fies documents of the known classes into their respective classes and also to reject docu- ments from unknown classes. This problem is called open (world) classification. This paper approaches the problem by reducing the open space risk while balancing the empirical risk. It proposes to use a new learning strategy, called center-based similarity (CBS) space learning (or CBS learning), to provide a novel solution to the problem. Extensive experi- ments across two datasets show that CBS learning gives promising results on multiclass open text classification compared to state-of- the-art baselines. 1 Introduction With the rapid growth of online information, text classifiers have become one of the most important tools for people to track and organize information. And the emergence of social media platforms has brought increasing diversity and dynamics to the Web. Many social science researchers rely on the collected online user generated content to carry out research on different social phenomenon. In this case, multiclass text classifiers are widely used to gather information of several topics of interest. However, most existing research on multiclass text classification makes the closed world assumption, meaning that all the test classes have been seen in training. However, in a more realistic scenario where people use a multiclass classifier to collect information of several topics from a data source that covers a much broader range of topics, it is normal to break the closed world assumption and to see the arrival of documents from unknown classes that have never been seen in training. In this case, a multiclass classifier should not always assign a document to one of the known classes. In- stead, it should identify unknown classes of docu- ments and label them as unknown or reject. This is called open (world) classification. More precisely, in the traditional multiclass classification setting, the learner assumes a fixed set of classes Y = {C1, C2, …, Cm}, and the task is to construct a 𝑚-class classifier using the training data. The resulting classifier is tested/applied on the data from only the m classes. While in open classification, we allow the classifier to predict la- bels/classes from the set of C1, C2, …, Cm, Cm+1 classes, where the (m+1)th class Cm+1 represents the unknown which covers documents of all unknown or unseen classes or topics. In other words, every test instance may be predicted to belong to either one of the known classes yi ∈ …

Content truncated to 500 words