Topic Modeling in Embedding Spaces

Topic Modeling in Embedding Spaces Adji B. Dieng Columbia University New York, NY, USA abd2141@columbia.edu Francisco J. R. Ruiz∗ DeepMind London, UK franrruiz@google.com David M. Blei Columbia University New York, NY, USA david.blei@columbia.edu Abstract Topic modeling analyzes documents to learn meaningful patterns of words. However, exist- ing topic models fail to learn interpretable topics when working with large and heavy- tailed vocabularies. To this end, we develop the embedded topic model (ETM), a generative model of documents that marries traditional topicmodelswith word embeddings. More spe- cifically, the ETM models each word with a categorical distribution whose natural param- eter is the inner product between the word’s embedding and an embedding of its assigned topic. To fit the ETM, we develop an effi- cient amortized variational inference algo- rithm. The ETM discovers interpretable topics even with large vocabularies that include rare words and stop words. It outperforms exist- ing document models, such as latent Dirichlet allocation, in terms of both topic quality and predictive performance. 1 Introduction Topic models are statistical tools for discovering the hidden semantic structure in a collection of documents (Blei et al., 2003; Blei, 2012). Topic models and their extensions have been applied to many fields, such as marketing, sociology, political science, and the digital humanities. Boyd-Graber et al. (2017) provide a review. Most topic models build on latent Dirichlet allocation (LDA) (Blei et al., 2003). LDA is a hierarchical probabilistic model that represents each topic as a distribution over terms and re- presents each document as a mixture of the top- ics. When fit to a collection of documents, the topics summarize their contents, and the topic ∗Work done while at Columbia University and the University of Cambridge. proportions provide a low-dimensional represen- tation of each document. LDA can be fit to large datasets of text by using variational inference and stochastic optimization (Hoffman et al., 2010, 2013). LDA is a powerful model and it is widely used. However, it suffers from a pervasive technical problem—it fails in the face of large vocabularies. Practitioners must severely prune their vocabular- ies in order to fit good topic models—namely, those that are both predictive and interpretable. This is typically done by removing the most and least frequent words. On large collections, this pruning may remove important terms and limit the scope of the models. The problem of topic modeling with large vocabularies has yet to be addressed in the research literature. In parallel with topic modeling came the idea of word embeddings. Research in word embeddings begins with the neural language model of Bengio et al. (2003), published in the same year and journal as Blei et al. (2003). Word embeddings eschew the ‘‘one-hot’’representation of words—a vocabulary-length vector of zeros with a single one—to learn a distributed representation, one where words with similar meanings are close in a lower-dimensional vector space (Rumelhart and Abrahamson, 1973; Bengio et al., 2006). As for topic models, researchers scaled up embedding methods to large datasets (Mikolov et al., 2013a,b; Pennington et al., …

Content truncated to 500 words