LogisticLDA: Regularizing Latent Dirichlet Allocation by Logistic Regression

LogisticLDA: Regularizing Latent Dirichlet Allocation by Logistic Regression⋆ Jia-Cheng Guo a, Bao-Liang Lu a,b, Zhiwei Li c, and Lei Zhang c aCenter for Brain-Like Computing and Machine Intelligence Department of Computer Science and Engineering bMOE-Microsoft Key Laboratory for Intelligent Computing and Intelligent Systems Shanghai Jiao Tong University 800 Dong Chuan Road, Shanghai 200040, China bllu@sjtu.edu.cn cMicrosoft Research Asia 49 Zhichun Road, Haidian District, Beijing 100080, China {zli, leizhang}@microsoft.com Abstract. We present in this paper a supervised topic model for multi-class classification problems. To incorporate supervisory information, we jointly model documents and their labels in a graphical model called LogisticLDA, which mathematically integrates a genera- tive model and a discriminative model in a principled way. By maximizing the posterior of document labels using logistic normal distributions, the model effectively incorporates the supervisory information to maximize inter-class distance in the topic space, while documents still enjoy the interchangeability characteristic for ease of inference. Experimental results on three benchmark datasets demonstrate that the model outperforms state-of-the-art supervised topic models. Compared with support vector machine, the model also achieves compara- ble performance, but meanwhile it discovers a topic space, which is valuable for dimension reduction, topic mining and document retrieval. Keywords: supervised topic model, text categorization, graph model, statistical learning 1 Introduction As an unsupervised method, Latent Dirichlet Allocation (LDA) (Blei et al., 2003) seeks to si- multaneously find a set of basis (i.e. topics) and embed documents to the latent space spanned by this basis. Due to its inherent capability of producing interpretable and semantically coherent topics, LDA has been widely used in text analysis and shown promising performance in tasks like topic mining, browsing, and accessing document similarity. In contrast, when LDA is applied to text classification tasks, it is often used as only a dimension reduction step to extract features for consecutive discriminative models (e.g. SVM). Because the objective of LDA (as well as other unsupervised topic models) is to infer the best set of latent topics that can explain the document collection rather than separate different classes, the training of LDA is actually independent to supervisory information. This paradigm greatly limits its applications in classification tasks. To address this issue, some topic models which integrate supervisory information have been proposed in recent years. Mimno and McCallum categorized these models to two types: upstream and downstream models. In “upstream” models, hidden topics are generated by conditioning on supervisory information(i.e. p(z|y), where z means a hidden topic and y means the category of ⋆ J.C. Guo and B.L. Lu are supported by the National Natural Science Foundation of China (Grant No. 60773090 and Grant No. 90820018), the National Basic Research Program of China (Grant No. 2009CB320901), and the National High-Tech Research Program of China (Grant No. 2008AA02Z315). Copyright 2009 by Jia-Cheng Guo, Bao-Liang Lu, Zhiwei Li, and Lei Zhang 160 23rd Pacific Asia Conference on Language, Information and Computation, pages 160–169 a document). Examples of upstream models include the Dirichlet Multinomial Regression model (DMR) (Mimno and McCallum, 2008), and the Theme Model …

Content truncated to 500 words