Practical Correlated Topic Modeling and Analysis via the Rectified Anchor Word Algorithm

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 4991–5001, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 4991 Practical Correlated Topic Modeling and Analysis via the Rectified Anchor Word Algorithm Moontae Lee1 Sungjun Cho2 David Bindel2 David Mimno2 1University of Illinois at Chicago, Microsoft Research at Redmond 2Cornell University moontae@uic.edu, {sc782,bindel,mimno}@cornell.edu Abstract Despite great scalability on large data and their ability to understand correlations between top- ics, spectral topic models have not been widely used due to the absence of reliability in real data and lack of practical implementations. This paper aims to solidify the foundations of spectral topic inference and provide a practical implementation for anchor-based topic mod- eling. Beginning with vocabulary curation, we scrutinize every single inference step with other viable options. We also evaluate our matrix-based approach against popular alterna- tives including a tensor-based spectral method as well as probabilistic algorithms. Our quanti- tative and qualitative experiments demonstrate the power of Rectified Anchor Word algorithm in various real datasets, providing a complete guide to practical correlated topic modeling. 1 Introduction Increasing access to massive data streams is useful only if it is equipped with proper tools to discover meaningful patterns. Topic models are capable of learning low-dimensional latent structures from groups of discrete observations, while being flexi- bly applicable to a wide range of modalities without human annotations. Users can assess consumer pro- files by collecting purchase habits (Reisenbichler and Reutterer, 2019), shared sentiments or topics among comments in social networks (Nguyen et al., 2015), hidden genres/preferences on movie or mu- sic consumption (Lee et al., 2015), and latent com- munities from network snapshots (Gerlach et al., 2018). For clarity this paper sticks to the standard terms — words, documents, and topics — but the concepts generalize to various applications beyond these examples. Traditional algorithms for topic modeling lack scalability. To learn quality topics, probabilistic algorithms such as Variational Inference (VI) or Markov Chain Monte Carlo (MCMC) require mul- tiple passes through the input dataset until con- vergence, and thus struggle to process millions of documents. Online or stochastic algorithms (Hoffman et al., 2010, 2013) achieve some scal- ability but at the cost of sacrificing the quality of topics. As a result, the basic Latent Dirichlet Al- location (Blei et al., 2003) is still prevalent for practitioners despite various recent advances (Sri- vastava and Sutton, 2017; Xun et al., 2017; Xu et al., 2018). However, topics are likely co-occur- ring terms in essence. Spectral methods explicitly construct word co-occurrence moments as statisti- cally unbiased estimators, providing alternatives to the probabilistic algorithms via moment-matching. Once the co-occurrence statistics is built with a sin- gle trivially parallelizable pass through the corpus, topic inference no longer needs to revisit individual training documents. The Anchor Word algorithm (Arora et al., 2012, 2013; Lee et al., 2015) and tensor decomposition algorithms (Anandkumar et al., 2012a,b, 2013) fac- torize the second and third-order co-occurrence be- …

Content truncated to 500 words