Shared Components Topic Models

2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 783–792, Montr´eal, Canada, June 3-8, 2012. c⃝2012 Association for Computational Linguistics Shared Components Topic Models Matthew R. Gormley Mark Dredze Benjamin Van Durme Jason Eisner Center for Language and Speech Processing Human Language Technology Center of Excellence Department of Computer Science Johns Hopkins University, Baltimore, MD {mrg,mdredze,vandurme,jason}@cs.jhu.edu Abstract With a few exceptions, extensions to latent Dirichlet allocation (LDA) have focused on the distribution over topics for each document. Much less attention has been given to the un- derlying structure of the topics themselves. As a result, most topic models generate topics in- dependently from a single underlying distri- bution and require millions of parameters, in the form of multinomial distributions over the vocabulary. In this paper, we introduce the Shared Components Topic Model (SCTM), in which each topic is a normalized product of a smaller number of underlying component dis- tributions. Our model learns these component distributions and the structure of how to com- bine subsets of them into topics. The SCTM can represent topics in a much more compact representation than LDA and achieves better perplexity with fewer parameters. 1 Introduction Topic models are probabilistic graphical models meant to capture the semantic associations underly- ing corpora. Since the introduction of latent Dirich- let allocation (LDA) (Blei et al., 2003), these mod- els have been extended to account for more complex distributions over topics, such as adding supervision (Blei and McAuliffe, 2007), non-parametric priors (Blei et al., 2004; Teh et al., 2006), topic correla- tions (Li and McCallum, 2006; Mimno et al., 2007; Blei and Lafferty, 2006) and sparsity (Williamson et al., 2010; Eisenstein et al., 2011). While much research has focused on modeling distributions over topics, less focus has been given to the makeup of the topics themselves. This emphasis leads us to find two problems with LDA and its vari- ants mentioned above: (1) independently generated topics and (2) overparameterized models. Independent Topics In the models above, the top- ics are modeled as independent draws from a single underlying distribution, typically a Dirichlet. This violates the topic modeling community’s intuition that these distributions over words are often related. As an example, consider a corpus that supports two related topics, baseball and hockey. These topics likely overlap in their allocation of mass to high probability words (e.g. team, season, game, play- ers), even though the two topics are unlikely to ap- pear in the same documents. When topics are gen- erated independently, the model does not provide a way to capture this sharing between related topics. Many extensions to LDA have addressed a related issue, LDA’s inability to model topic correlation,1 by changing the distributions over topics (Blei and Lafferty, 2006; Li and McCallum, 2006; Mimno et al., 2007; Paisley et al., 2011). Yet, none of these change the underlying structure of the topic’s distri- butions over words. Overparameterization Topics are most often parameterized as multinomial distributions over words: increasing the topics …

Content truncated to 500 words