Bayesian Checking for Topic Models

Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 227–237, Edinburgh, Scotland, UK, July 27–31, 2011. c⃝2011 Association for Computational Linguistics Bayesian Checking for Topic Models David Mimno Department of Computer Science Princeton University Princeton, NJ 08540 mimno@cs.princeton.edu David Blei Department of Computer Science Princeton University Princeton, NJ 08540 blei@cs.princeton.edu Abstract Real document collections do not fit the inde- pendence assumptions asserted by most statis- tical topic models, but how badly do they vi- olate them? We present a Bayesian method for measuring how well a topic model fits a corpus. Our approach is based on posterior predictive checking, a method for diagnosing Bayesian models in user-defined ways. Our method can identify where a topic model fits the data, where it falls short, and in which di- rections it might be improved. 1 Introduction Probabilistic topic models are a suite of machine learning algorithms that decompose a corpus into a set of topics and represent each document with a subset of those topics. The inferred topics often cor- respond with the underlying themes of the analyzed collection, and the topic modeling algorithm orga- nizes the documents according to those themes. Most topic models are evaluated by their predic- tive performance on held out data. The idea is that topic models are fit to maximize the likelihood (or posterior probability) of a collection of documents, and so a good model is one that assigns high likeli- hood to a held out set (Blei et al., 2003; Wallach et al., 2009). But this evaluation is not in line with how topic models are frequently used. Topic mod- els seem to capture the underlying themes of a collection—indeed the monicker “topic model” is retrospective—and so we expect that these themes are useful for exploring, summarizing, and learning about its documents (Mimno and McCallum, 2007; Chang et al., 2009). In such exploratory data anal- ysis, however, we are not concerned with the fit to held out data. In this paper, we develop and study new methods for evaluating topic models. Our methods are based on posterior predictive checking, which is a model diagnosis technique from Bayesian statistics (Rubin, 1984; Gelman et al., 1996). The goal of a posterior predictive check (PPC) is to assess the validity of a Bayesian model without requiring a specific alterna- tive model. Given data, we first compute a posterior distribution over the latent variables. Then, we esti- mate the probability of the observed data under the data-generating distribution that is induced by the posterior (the “posterior predictive distribution”). A data set that is unlikely calls the model into ques- tion, and consequently the posterior. PPCs can show where the model fits and doesn’t fit the observations. They can help identify the parts of the posterior that are worth exploring. The key to a posterior predictive check is the dis- crepancy function. This is a function of the data that measures a property of the model which is impor- tant to capture. While the model is …

Content truncated to 500 words