Anchors Regularized: Adding Robustness and Extensibility to Scalable Topic-Modeling Algorithms

Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, pages 359–369, Baltimore, Maryland, USA, June 23-25 2014. c⃝2014 Association for Computational Linguistics Anchors Regularized: Adding Robustness and Extensibility to Scalable Topic-Modeling Algorithms Thang Nguyen iSchool and UMIACS, University of Maryland and National Library of Medicine, National Institutes of Health daithang@umiacs.umd.edu Yuening Hu Computer Science University of Maryland ynhu@cs.umd.edu Jordan Boyd-Graber iSchool and UMIACS University of Maryland jbg@umiacs.umd.edu Abstract Spectral methods offer scalable alternatives to Markov chain Monte Carlo and expec- tation maximization. However, these new methods lack the rich priors associated with probabilistic models. We examine Arora et al.’s anchor words algorithm for topic mod- eling and develop new, regularized algo- rithms that not only mathematically resem- ble Gaussian and Dirichlet priors but also improve the interpretability of topic models. Our new regularization approaches make these efficient algorithms more flexible; we also show that these methods can be com- bined with informed priors. 1 Introduction Topic models are of practical and theoretical inter- est. Practically, they have been used to understand political perspective (Paul and Girju, 2010), im- prove machine translation (Eidelman et al., 2012), reveal literary trends (Jockers, 2013), and under- stand scientific discourse (Hall et al., 2008). The- oretically, their latent variable formulation has served as a foundation for more robust models of other linguistic phenomena (Brody and Lapata, 2009). Modern topic models are formulated as a la- tent variable model. Like hidden Markov mod- els (Rabiner, 1989, HMM), each token comes from one of K unknown distributions. Unlike a HMM, topic models assume that each document is an ad- mixture of these hidden components called topics. Posterior inference discovers the hidden variables that best explain a dataset. Typical solutions use MCMC (Griffiths and Steyvers, 2004) or variational EM (Blei et al., 2003), which can be viewed as local optimization: searching for the latent variables that maximize the data likelihood. An exciting vein of new research provides provable polynomial-time alternatives. These ap- proaches provide solutions to hidden Markov mod- els (Anandkumar et al., 2012), mixture mod- els (Kannan et al., 2005), and latent variable gram- mars (Cohen et al., 2013). The key insight is not to directly optimize observation likelihood but to in- stead discover latent variables that can reconstruct statistics of the assumed generative model. Unlike search-based methods, which can be caught in lo- cal minima, these techniques are often guaranteed to find global optima. These general techniques can be improved by making reasonable assumptions about the models. For example, Arora et al. (2012b)’s approach for in- ference in topic models assume that each topic has a unique “anchor” word (thus, we call this approach anchor). This approach is fast and effective; be- cause it only uses word co-occurrence information, it can scale to much larger datasets than MCMC or EM alternatives. We review the anchor method in Section 2. Despite their advantages, these techniques are not a panacea. They do not accommodate the rich priors that modelers have come to expect. Priors can …

Content truncated to 500 words