Optimizing Semantic Coherence in Topic Models

Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 262–272, Edinburgh, Scotland, UK, July 27–31, 2011. c⃝2011 Association for Computational Linguistics Optimizing Semantic Coherence in Topic Models David Mimno Princeton University Princeton, NJ 08540 mimno@cs.princeton.edu Hanna M. Wallach University of Massachusetts, Amherst Amherst, MA 01003 wallach@cs.umass.edu Edmund Talley Miriam Leenders National Institutes of Health Bethesda, MD 20892 {talleye,leenderm}@ninds.nih.gov Andrew McCallum University of Massachusetts, Amherst Amherst, MA 01003 mccallum@cs.umass.edu Abstract Latent variable models have the potential to add value to large document collections by discovering interpretable, low-dimensional subspaces. In order for people to use such models, however, they must trust them. Un- fortunately, typical dimensionality reduction methods for text, such as latent Dirichlet al- location, often produce low-dimensional sub- spaces (topics) that are obviously flawed to human domain experts. The contributions of this paper are threefold: (1) An analysis of the ways in which topics can be flawed; (2) an au- tomated evaluation metric for identifying such topics that does not rely on human annotators or reference collections outside the training data; (3) a novel statistical topic model based on this metric that significantly improves topic quality in a large-scale document collection from the National Institutes of Health (NIH). 1 Introduction Statistical topic models such as latent Dirichlet al- location (LDA) (Blei et al., 2003) provide a pow- erful framework for representing and summarizing the contents of large document collections. In our experience, however, the primary obstacle to accep- tance of statistical topic models by users the outside machine learning community is the presence of poor quality topics. Topics that mix unrelated or loosely- related concepts substantially reduce users’ confi- dence in the utility of such automated systems. In general, users prefer models with larger num- bers of topics because such models have greater res- olution and are able to support finer-grained distinc- tions. Unfortunately, we have observed that there is a strong relationship between the size of topics and the probability of topics being nonsensical as judged by domain experts: as the number of topics increases, the smallest topics (number of word to- kens assigned to each topic) are almost always poor quality. The common practice of displaying only a small number of example topics hides the fact that as many as 10% of topics may be so bad that they can- not be shown without reducing users’ confidence. The evaluation of statistical topic models has tra- ditionally been dominated by either extrinsic meth- ods (i.e., using the inferred topics to perform some external task such as information retrieval (Wei and Croft, 2006)) or quantitative intrinsic methods, such as computing the probability of held-out doc- uments (Wallach et al., 2009). Recent work has focused on evaluation of topics as semantically- coherent concepts. For example, Chang et al. (2009) found that the probability of held-out documents is not always a good predictor of human judgments. Newman et al. (2010) showed that an automated evaluation metric based on word co-occurrence statistics gathered from Wikipedia could predict hu- man …

Content truncated to 500 words