Exploring Topic Coherence over Many Models and Many Topics

Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 952–961, Jeju Island, Korea, 12–14 July 2012. c⃝2012 Association for Computational Linguistics Exploring Topic Coherence over many models and many topics Keith Stevens1,2 Philip Kegelmeyer3 David Andrzejewski2 David Buttler2 1University of California Los Angeles; Los Angeles , California, USA 2Lawrence Livermore National Lab; Livermore, California, USA 3Sandia National Lab; Livermore, California, USA {stevens35,andrzejewski1,buttler1}@llnl.gov wpk@sandia.gov Abstract We apply two new automated semantic eval- uations to three distinct latent topic models. Both metrics have been shown to align with human evaluations and provide a balance be- tween internal measures of information gain and comparisons to human ratings of coher- ent topics. We improve upon the measures by introducing new aggregate measures that allows for comparing complete topic models. We further compare the automated measures to other metrics for topic models, compar- ison to manually crafted semantic tests and document classification. Our experiments re- veal that LDA and LSA each have different strengths; LDA best learns descriptive topics while LSA is best at creating a compact se- mantic representation of documents and words in a corpus. 1 Introduction Topic models learn bags of related words from large corpora without any supervision. Based on the words used within a document, they mine topic level relations by assuming that a single document cov- ers a small set of concise topics. Once learned, these topics often correlate well with human con- cepts. For example, one model might produce topics that cover ideas such as government affairs, sports, and movies. With these unsupervised methods, we can extract useful semantic information in a variety of tasks that depend on identifying unique topics or concepts, such as distributional semantics (Jurgens and Stevens, 2010), word sense induction (Van de Cruys and Apidianaki, 2011; Brody and Lapata, 2009), and information retrieval (Andrzejewski and Buttler, 2011). When using a topic model, we are primarily con- cerned with the degree to which the learned top- ics match human judgments and help us differen- tiate between ideas. But until recently, the evalua- tion of these models has been ad hoc and applica- tion specific. Evaluations have ranged from fully automated intrinsic evaluations to manually crafted extrinsic evaluations. Previous extrinsic evaluations have used the learned topics to compactly represent a small fixed vocabulary and compared this distribu- tional space to human judgments of similarity (Jur- gens and Stevens, 2010). But these evaluations are hand constructed and often costly to perform for domain-specific topics. Conversely, intrinsic mea- sures have evaluated the amount of information en- coded by the topics, where perplexity is one com- mon example(Wallach et al., 2009), however, Chang et al. (2009) found that these intrinsic measures do not always correlate with semantically interpretable topics. Furthermore, few evaluations have used the same metrics to compare distinct approaches such as Latent Dirichlet Allocation (LDA) (Blei et al., 2003), Latent Semantic Analysis (LSA) (Landauer and Dutnais, 1997), and Non-negative Matrix Fac- torization (NMF) (Lee and Seung, 2000). …

Content truncated to 500 words