Bayesian Document Generative Model with Explicit Multiple Topics

Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pp. 421–429, Prague, June 2007. c⃝2007 Association for Computational Linguistics Bayesian Document Generative Model with Explicit Multiple Topics Issei Sato Graduate School of Information Science and Technology, The University of Tokyo sato@r.dl.itc.u-tokyo.ac.jp Hiroshi Nakagawa Information Technology Center, The University of Tokyo nakagawa@dl.itc.u-tokyo.ac.jp Abstract In this paper, we proposed a novel prob- abilistic generative model to deal with ex- plicit multiple-topic documents: Parametric Dirichlet Mixture Model(PDMM). PDMM is an expansion of an existing probabilis- tic generative model: Parametric Mixture Model(PMM) by hierarchical Bayes model. PMM models multiple-topic documents by mixing model parameters of each single topic with an equal mixture ratio. PDMM models multiple-topic documents by mix- ing model parameters of each single topic with mixture ratio following Dirichlet dis- tribution. We evaluate PDMM and PMM by comparing F-measures using MEDLINE corpus. The evaluation showed that PDMM is more effective than PMM. 1 Introduction Documents, such as those seen on Wikipedia and Folksonomy, have tended to be assigned with ex- plicit multiple topics. In this situation, it is impor- tant to analyze a linguistic relationship between doc- uments and the assigned multiple topics . We at- tempt to model this relationship with a probabilistic generative model. A probabilistic generative model for documents with multiple topics is a probability model of the process of generating documents with multiple topics. By focusing on modeling the gener- ation process of documents and the assigned multi- ple topics, we can extract specific properties of doc- uments and the assigned multiple topics. The model can also be applied to a wide range of applications such as automatic categorization for multiple topics, keyword extraction and measuring document simi- larity, for example. A probabilistic generative model for documents with multiple topics is categorized into the following two models. One model assumes a topic as a latent topic. We call this model the latent-topic model. The other model assumes a topic as an explicit topic. We call this model the explicit-topic model. In a latent-topic model, a latent topic indicates not a concrete topic but an underlying implicit topic of documents. Obviously this model uses an unsupervised learning algorithm. Representa- tive examples of this kind of model are Latent Dirichlet Allocation(LDA)(D.M.Blei et al., 2001; D.M.Blei et al., 2003) and Hierarchical Dirichlet Process(HDP)(Y.W.Teh et al., 2003). In an explicit-topic model, an explicit topic indi- cates a concrete topic such as economy or sports, for example. A learning algorithm for this model is a supervised learning algorithm. That is, an explicit topic model learns model parameter using a training data set of tuples such as (documents, topics). Rep- resentative examples of this model are Parametric Mixture Models(PMM1 and PMM2)(Ueda, N. and Saito, K., 2002a; Ueda, N. and Saito, K., 2002b). In the remainder of this paper, PMM indicates PMM1 because PMM1 is more effective than PMM2. In this paper, we focus on the explicit topic model. In particular, we propose a novel model …

Content truncated to 500 words