Efficient Methods for Inferring Large Sparse Topic Hierarchies

Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 774–784, Beijing, China, July 26-31, 2015. c⃝2015 Association for Computational Linguistics Efficient Methods for Inferring Large Sparse Topic Hierarchies Doug Downey, Chandra Sekhar Bhagavatula, Yi Yang Electrical Engineering and Computer Science Northwestern University ddowney@eecs.northwestern.edu,{csb,yiyang}@u.northwestern.edu Abstract Latent variable topic models such as La- tent Dirichlet Allocation (LDA) can dis- cover topics from text in an unsupervised fashion. However, scaling the models up to the many distinct topics exhibited in modern corpora is challenging. “Flat” topic models like LDA have difficulty modeling sparsely expressed topics, and richer hierarchical models become compu- tationally intractable as the number of top- ics increases. In this paper, we introduce efficient meth- ods for inferring large topic hierarchies. Our approach is built upon the Sparse Backoff Tree (SBT), a new prior for la- tent topic distributions that organizes the latent topics as leaves in a tree. We show how a document model based on SBTs can effectively infer accurate topic spaces of over a million topics. We introduce a collapsed sampler for the model that ex- ploits sparsity and the tree structure in or- der to make inference efficient. In exper- iments with multiple data sets, we show that scaling to large topic spaces results in much more accurate models, and that SBT document models make use of large topic spaces more effectively than flat LDA. 1 Introduction Latent variable topic models, such as Latent Dirichlet Allocation (Blei et al., 2003), are popu- lar approaches for automatically discovering top- ics in document collections. However, learning models that capture the large numbers of distinct topics expressed in today’s corpora is challenging. While efficient methods for learning large topic models have been developed (Li et al., 2014; Yao et al., 2009; Porteous et al., 2008), these methods have focused on “flat” topic models such as LDA. Flat topic models over large topic spaces are prone to overfitting: even in a Web-scale corpus, some words are expressed rarely, and many documents are brief. Inferring a large topic distribution for each word and document given such sparse data is challenging. As a result, LDA models in prac- tice tend to consider a few thousand topics at most, even when training on billions of words (Mimno et al., 2012). A promising alternative to flat topic models is found in recent hierarchical topic models (Paisley et al., 2015; Blei et al., 2010; Li and McCallum, 2006; Wang et al., 2013; Kim et al., 2013; Ahmed et al., 2013). Topics of words and documents can be naturally arranged into hierarchies. For exam- ple, an article on the topic of the Chicago Bulls is also relevant to the more general topics of NBA, Basketball, and Sports. Hierarchies can combat data sparsity: if data is too sparse to place the term “Pau Gasol” within the Chicago Bulls topic, perhaps it can be appropriately modeled at some- what less precision within the Basketball topic. …

Content truncated to 500 words