A Pseudo Label based Dataless Naive Bayes Algorithm for Text Classification with Seed Words

Proceedings of the 27th International Conference on Computational Linguistics, pages 1908–1917 Santa Fe, New Mexico, USA, August 20-26, 2018. 1908 A Pseudo Label based Dataless Naive Bayes Algorithm for Text Classification with Seed Words Ximing Li, Bo Yang∗ College of Computer Science and Technology, Jilin University, China Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, China liximing86@gmail.com; ybo@jlu.edu.cn Abstract Traditional supervised text classifiers require a large number of manually labeled documents, which are often expensive to obtain. Recently, dataless text classification has attracted more attention, since it only requires very few seed words of categories that are much cheaper. In this paper, we develop a pseudo-label based dataless Naive Bayes (PL-DNB) classifier with seed words. We initialize pseudo-labels for each document using seed word occurrences, and em- ploy the expectation maximization algorithm to train PL-DNB in a semi-supervised manner. The pseudo-labels are iteratively updated using a mixture of seed word occurrences and estimations of label posteriors. To avoid noisy pseudo-labels, we also consider the information of near- est neighboring documents in the pseudo-label update step, i.e., preserving local neighborhood structure of documents. We empirically show that PL-DNB outperforms traditional dataless text classification algorithms with seed words. Especially, PL-DNB performs well on the imbalanced dataset. 1 Introduction Automatic text classification is one of the most popular directions in the machine learning community. A typical procedure for creating a classifier in supervised learning consists of two steps: (1) Given a collection of text documents, human experts define a number of category labels, and then manually assign these pre-defined labels to documents. We refer to these labeled documents as training dataset; (2) A supervised learning algorithm, e.g., support vector machines (SVMs), is trained on the training dataset, outputting a classifier for predicting future documents. Manually labeling documents, i.e., step (1), is very expensive and time-consuming. Unfortunately, supervised learning algorithms often require massive labeled documents to avoid learning issues such as overfitting (Cawley and Talbot, 2010). A common way to reduce the labeling effort is developing semi- supervised learning algorithms, where one trains text classifiers on a mixture collection of a few labeled documents and a larger number of unlabeled documents (Nigam et al., 2000; Hu et al., 2017). However, semi-supervised learning still requires labeled documents, and manually labeling a small number of documents remains very expensive in many real world applications. Recently, researchers have proposed a number of dataless text classification algorithms (Liu et al., 2004; Chang et al., 2008; Downey and Etzioni, 2008; Druck et al., 2008; Hingmire et al., 2013; Hing- mire and Chakraborti, 2014; Chen et al., 2015; Li et al., 2016), which do not require labeled documents as training instances. Instead, they train text classifiers using unlabeled documents with seed words, i.e., the selected representative words for categories. Actually, manually choosing seed words is significantly cheaper than labeling documents (Raghavan et al., 2006; Druck et al., 2008), saving many human effort- s. This kind of dataless algorithm has empirically achieved promising classification results, and it …

Content truncated to 500 words