Negative Training Data Can be Harmful to Text Classification

Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 218–228, MIT, Massachusetts, USA, 9-11 October 2010. c⃝2010 Association for Computational Linguistics Negative Training Data can be Harmful to Text Classification Xiao-Li Li Bing Liu See-Kiong Ng Institute for Infocomm Research University of Illinois at Chicago Institute for Infocomm Research 1 Fusionopolis Way #21-01, Connexis Singapore 138632 851 South Morgan Street, Chicago, IL 60607-7053, USA 1 Fusionopolis Way #21-01, Connexis Singapore 138632 xlli@i2r.a-star.edu.sg liub@cs.uic.edu skng@i2r.a-star.edu.sg Abstract This paper studies the effects of training data on binary text classification and postulates that negative training data is not needed and may even be harmful for the task. Traditional binary classification involves building a clas- sifier using labeled positive and negative training examples. The classifier is then ap- plied to classify test instances into positive and negative classes. A fundamental assump- tion is that the training and test data are iden- tically distributed. However, this assumption may not hold in practice. In this paper, we study a particular problem where the positive data is identically distributed but the negative data may or may not be so. Many practical text classification and retrieval applications fit this model. We argue that in this setting nega- tive training data should not be used, and that PU learning can be employed to solve the problem. Empirical evaluation has been con- ducted to support our claim. This result is im- portant as it may fundamentally change the current binary classification paradigm. 1 Introduction Text classification is a well-studied problem in machine learning, natural language processing, and information retrieval. To build a text classifier, a set of training documents is first labeled with pre- defined classes. Then, a supervised machine learn- ing algorithm (e.g., Support Vector Machines (SVM), naïve Bayesian classifier (NB)) is applied to the training examples to build a classifier that is subsequently employed to assign class labels to the instances in the test set. In this paper, we focus on binary text classification with two classes (i.e. pos- itive and negative classes). Most learning methods assume that the training and test data have identical distributions. However, this assumption may not hold in practice, i.e., the training and the test distributions can be different. The problem is called covariate shift or sample selection bias (Heckman 1979; Shimodaira 2000; Zadrozny 2004; Huang et al. 2007; Sugiyama et al. 2008; Bickel et al. 2009). In general, this problem is not solvable because the two distributions can be arbitrarily far apart from each other. Various as- sumptions were made to solve special cases of the problem. One main assumption was that the condi- tional distribution of the class given an instance is the same over the training and test sets (Shimodai- ra 2000; Huang et al. 2007; Bickel et al. 2009). In this paper, we study another special case of the problem in which the positive training and test samples have identical distributions, but the nega- tive training and test samples may have different distributions. We believe …

Content truncated to 500 words