Semantic Naïve Bayes Classifier for Document Classification

International Joint Conference on Natural Language Processing, pages 1117–1123, Nagoya, Japan, 14-18 October 2013. Semantic Naïve Bayes Classifier for Document Classification How Jing, Yu Tsao Research Center for Information Technology Innovation Academia Sinica, Taipei, Taiwan {yu.tsao}@citi.sinica.edu.tw Kuan-Yu Chen, Hsin-Min Wang Institute of Information Science Academia Sinica, Taipei, Taiwan {kychen, whm}@iis.sinica.edu.tw Abstract In this paper, we propose a semantic naïve Bayes classifier (SNBC) to improve the conventional naïve Bayes classifier (NBC) by incorporating “document-level” semantic information for document classification (DC). To capture the semantic information from each document, we develop semantic feature extraction and modeling algorithms. For semantic feature extraction, we first apply a log-Bilinear document modeling (LBDM) algorithm to transform each word into a semantic vector, and then apply principal component analysis (PCA) to the semantic space formed by the word vectors to extract a set of semantic features for each document. For semantic modeling, a semantic model is constructed using the semantic features of the training documents. In the testing phase, SNBC systematically integrates the semantic model and the conventional NBC to perform DC. The results of experiments on the 20 News-groups and WebKB datasets confirm that, with the semantic score, SNBC consistently outperforms NBC with various language modeling approaches. 1 Introduction Document classification (DC) is an important task in the information retrieval (IR) and natural language processing (NLP) areas. In recent years, many approaches have been developed for DC. Among them, a category of approaches views DC as a traditional ranking problem. These approaches first represent a document with a feature vector, known as the vector space model (VSM), and then machine learning algorithms can be applied to accomplish classification. Notable examples belonging to this category include support vector machine (Joachims, 1998), decision tree (Comite et al., 2003), logistic regression (Genkin et al., 2005), and k-nearest neighbor (Kwon and Lee, 2003). Another successful category of approaches is based on the naïve Bayes classifier (NBC). NBC assumes that a document is generated from a probabilistic model. In the offline training process, the model parameters are estimated from a set of training documents. When performing DC, NBC calculates a conditional probability  Pcd | (the posterior probability that document d belongs to class c) and classifies the test document to the class that gives the highest  Pcd | . When calculating  Pcd | , NBC makes word independence (bag-of-words) assumptions and decomposes  Pcd | to a product of individual word probabilities. These word probabilities are usually characterized by a statistical language model. In practice, unigram language modeling (ULM) is a popular choice due to its effectiveness and computational efficiency (Peng and Schuurmans, 2003; Bai and Nie, 2004; Wu and Wang, 2012). However, since NBC with ULM only considers frequencies of words occurring in a class, it may suffer from the problems of data sparseness and word usage diversity, leading to performance degradations for DC. To deal with the data sparseness (zero probability) problem of ULM, many smoothing techniques, such as Laplace (Bai and Nie, 2004) and Jelinek-Mercer …

Content truncated to 500 words