Automatic Evaluation of Topic Coherence

Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the ACL, pages 100–108, Los Angeles, California, June 2010. c⃝2010 Association for Computational Linguistics Automatic Evaluation of Topic Coherence David Newman,♠♣ Jey Han Lau,♥ Karl Grieser♦, and Timothy Baldwin,♠♥ ♠ NICTA Victoria Research Laboratory, Australia ♣ Dept of Computer Science, University of California, Irvine ♥ Dept of Computer Science and Software Engineering, University of Melbourne, Australia ♦ Dept of Information Systems, University of Melbourne, Australia newman@uci.edu, depthchargex@gmail.com, kgrieser@csse.unimelb.edu.au, tb@ldwin.net Abstract This paper introduces the novel task of topic coherence evaluation, whereby a set of words, as generated by a topic model, is rated for coherence or interpretability. We apply a range of topic scoring models to the evaluation task, drawing on WordNet, Wikipedia and the Google search engine, and existing research on lexical similarity/relatedness. In compar- ison with human scores for a set of learned topics over two distinct datasets, we show a simple co-occurrence measure based on point- wise mutual information over Wikipedia data is able to achieve results for the task at or nearing the level of inter-annotator correla- tion, and that other Wikipedia-based lexical relatedness methods also achieve strong re- sults. Google produces strong, if less consis- tent, results, while our results over WordNet are patchy at best. 1 Introduction There has traditionally been strong interest within computational linguistics in techniques for learning sets of words (aka topics) which capture the latent semantics of a document or document collection, in the form of methods such as latent semantic analysis (Deerwester et al., 1990), probabilistic latent seman- tic analysis (Hofmann, 2001), random projection (Widdows and Ferraro, 2008), and more recently, la- tent Dirichlet allocation (Blei et al., 2003; Griffiths and Steyvers, 2004). Such methods have been suc- cessfully applied to a myriad of tasks including word sense discrimination (Brody and Lapata, 2009), doc- ument summarisation (Haghighi and Vanderwende, 2009), areal linguistic analysis (Daume III, 2009) and text segmentation (Sun et al., 2008). In each case, extrinsic evaluation has been used to demon- strate the effectiveness of the learned topics in the application domain, but standardly, no attempt has been made to perform intrinsic evaluation of the top- ics themselves, either qualitatively or quantitatively. In machine learning, on the other hand, researchers have modified and extended topic models in a vari- ety of ways, and evaluated intrinsically in terms of model perplexity (Wallach et al., 2009), but there has been less effort on qualitative understanding of the semantic nature of the learned topics. This research seeks to fill the gap between topic evaluation in computational linguistics and machine learning, in developing techniques to perform intrin- sic qualitative evaluation of learned topics. That is, we develop methods for evaluating the qual- ity of a given topic, in terms of its coherence to a human. After learning topics from a collection of news articles and a collection of books, we ask humans to decide whether individual learned top- ics are coherent, in terms of their interpretability and association …

Content truncated to 500 words