Scalable Collapsed Inference for High-Dimensional Topic Models

Proceedings of NAACL-HLT 2019, pages 2836–2845 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 2836 Scalable Collapsed Inference for High-Dimensional Topic Models Rashidul Islam and James Foulds Department of Information Systems University of Maryland, Baltimore County {islam.rashidul,jfoulds}@umbc.edu Abstract The bigger the corpus, the more topics it can potentially support. To truly make full use of massive text corpora, a topic model inference algorithm must therefore scale efficiently in 1) documents and 2) topics, while 3) achiev- ing accurate inference. Previous methods have achieved two out of three of these criteria si- multaneously, but never all three at once. In this paper, we develop an online inference algorithm for topic models which leverages stochasticity to scale well in the number of documents, sparsity to scale well in the num- ber of topics, and which operates in the col- lapsed representation of the topic model for improved accuracy and run-time performance. We use a Monte Carlo inner loop in the on- line setting to approximate the collapsed vari- ational Bayes updates in a sparse and efficient way, which we accomplish via the Metropolis- Hastings Walker method. We showcase our algorithm on LDA and the recently proposed mixed membership skip-gram topic model. Our method requires only amortized O(kd) computation per word token instead of O(K) operations, where the number of topics occur- ring for a particular document kd ≪ the total number of topics in the corpus K, to converge to a high-quality solution. 1 Introduction Topic models are powerful tools for analyzing to- day’s massive, constantly expanding digital text information by representing high-dimensional data in a low-dimensional subspace. We can re- cover the main themes of a corpus by using topic models such as latent Dirichlet allocation (LDA) to organize, understand, search, and explore the documents (Blei et al., 2003). Traditional LDA inference techniques such as variational Bayes and collapsed Gibbs sampling do not readily scale to corpora containing mil- lions of documents. To scale up inference, the main approaches are distributed algorithms (New- man et al., 2008) and stochastic algorithms (Hoff- man et al., 2010, 2013). Stochastic algorithms, such as stochastic variational inference (SVI), op- erate in an online fashion, and hence do not need to see all of the documents before updating the topics, so they can be applied to corpora of any size, without expensive distributed hardware (Hoffman et al., 2010). The “collapsed” represen- tation of topic models is also frequently important, as it leads to faster convergence, efficient updates, and lower variance in estimation (Griffiths and Steyvers, 2004). The stochastic collapsed vari- ational Bayesian inference (SCVB0) algorithm, proposed by (Foulds et al., 2013), combines the benefits of stochastic and collapsed inference. Larger corpora typically support more topics, which brings the additional efficiency challenge of training a larger model (Mimno et al., 2012). This challenge has been addressed by exploiting sparsity to perform updates in time sublinear in the number of topics. A sparse variant of the SVI algorithm for LDA, SSVI, proposed by …

Content truncated to 500 words