Sometimes Average is Best: The Importance of Averaging for Prediction using MCMC Inference in Topic Modeling

Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1752–1757, October 25-29, 2014, Doha, Qatar. c⃝2014 Association for Computational Linguistics Sometimes Average is Best: The Importance of Averaging for Prediction using MCMC Inference in Topic Modeling Viet-An Nguyen Computer Science University of Maryland College Park, MD vietan@cs.umd.edu Jordan Boyd-Graber Computer Science University of Colorado Boulder, CO jbg@boydgraber.org Philip Resnik Linguistics and UMIACS University of Maryland College Park, MD resnik@umd.edu Abstract Markov chain Monte Carlo (MCMC) approxi- mates the posterior distribution of latent vari- able models by generating many samples and averaging over them. In practice, however, it is often more convenient to cut corners, using only a single sample or following a suboptimal averaging strategy. We systematically study dif- ferent strategies for averaging MCMC samples and show empirically that averaging properly leads to significant improvements in prediction. 1 Introduction Probabilistic topic models are powerful methods to un- cover hidden thematic structures in text by projecting each document into a low dimensional space spanned by a set of topics, each of which is a distribution over words. Topic models such as latent Dirichlet alloca- tion (Blei et al., 2003, LDA) and its extensions discover these topics from text, which allows for effective ex- ploration, analysis, and summarization of the otherwise unstructured corpora (Blei, 2012; Blei, 2014). In addition to exploratory data analysis, a typical goal of topic models is prediction. Given a set of unanno- tated training data, unsupervised topic models try to learn good topics that can generalize to unseen text. Supervised topic models jointly capture both the text and associated metadata such as a continuous response variable (Blei and McAuliffe, 2007; Zhu et al., 2009; Nguyen et al., 2013), single label (Rosen-Zvi et al., 2004; Lacoste-Julien et al., 2008; Wang et al., 2009) or multiple labels (Ramage et al., 2009; Ramage et al., 2011) to predict metadata from text. Probabilistic topic modeling requires estimating the posterior distribution. Exact computation of the poste- rior is often intractable, which motivates approximate inference techniques (Asuncion et al., 2009). One popu- lar approach is Markov chain Monte Carlo (MCMC), a class of inference algorithms to approximate the target posterior distribution. To make prediction, MCMC al- gorithms generate samples on training data to estimate corpus-level latent variables, and use them to generate samples to estimate document-level latent variables for test data. The underlying theory requires averaging on both training and test samples, but in practice it is often convenient to cut corners: either skip averaging entirely by using just the values of the last sample or use a single training sample and average over test samples. We systematically study non-averaging and averaging strategies when performing predictions using MCMC in topic modeling (Section 2). Using popular unsupervised (LDA in Section 3) and supervised (SLDA in Section 4) topic models via thorough experimentation, we show empirically that cutting corners on averaging leads to consistently poorer prediction. 2 Learning and Predicting with MCMC While reviewing all of MCMC is beyond the scope …

Content truncated to 500 words