Gaussian LDA for Topic Models with Word Embeddings

Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 795–804, Beijing, China, July 26-31, 2015. c⃝2015 Association for Computational Linguistics Gaussian LDA for Topic Models with Word Embeddings Rajarshi Das*, Manzil Zaheer*, Chris Dyer School of Computer Science Carnegie Mellon University Pittsburgh, PA, 15213, USA {rajarshd, manzilz, cdyer} @cs.cmu.edu Abstract Continuous space word embeddings learned from large, unstructured corpora have been shown to be effective at cap- turing semantic regularities in language. In this paper we replace LDA’s param- eterization of “topics” as categorical distributions over opaque word types with multivariate Gaussian distributions on the embedding space. This encourages the model to group words that are a priori known to be semantically related into topics. To perform inference, we introduce a fast collapsed Gibbs sampling algorithm based on Cholesky decom- positions of covariance matrices of the posterior predictive distributions. We fur- ther derive a scalable algorithm that draws samples from stale posterior predictive distributions and corrects them with a Metropolis–Hastings step. Using vectors learned from a domain-general corpus (English Wikipedia), we report results on two document collections (20-newsgroups and NIPS). Qualitatively, Gaussian LDA infers different (but still very sensible) topics relative to standard LDA. Quantita- tively, our technique outperforms existing models at dealing with OOV words in held-out documents. 1 Introduction Latent Dirichlet Allocation (LDA) is a Bayesian technique that is widely used for inferring the topic structure in corpora of documents. It con- ceives of a document as a mixture of a small num- ber of topics, and topics as a (relatively sparse) dis- tribution over word types (Blei et al., 2003). These priors are remarkably effective at producing useful *Both student authors had equal contribution. results. However, our intuitions tell us that while documents may indeed be conceived of as a mix- ture of topics, we should further expect topics to be semantically coherent. Indeed, standard human evaluations of topic modeling performance are de- signed to elicit assessment of semantic coherence (Chang et al., 2009; Newman et al., 2009). How- ever, this prior preference for semantic coherence is not encoded in the model, and any such obser- vation of semantic coherence found in the inferred topic distributions is, in some sense, accidental. In this paper, we develop a variant of LDA that oper- ates on continuous space embeddings of words— rather than word types—to impose a prior expec- tation for semantic coherence. Our approach re- places the opaque word types usually modeled in LDA with continuous space embeddings of these words, which are generated as draws from a mul- tivariate Gaussian. How does this capture our preference for se- mantic coherence? Word embeddings have been shown to capture lexico-semantic regularities in language: words with similar syntactic and seman- tic properties are found to be close to each other in the embedding space (Agirre et al., 2009; Mikolov et al., 2013). Since Gaussian distributions capture a notion of centrality in space, and semantically related words …

Content truncated to 500 words