Machine Reading Tea Leaves: Automatically Evaluating Topic Coherence and Topic Model Quality

Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 530–539, Gothenburg, Sweden, April 26-30 2014. c⃝2014 Association for Computational Linguistics Machine Reading Tea Leaves: Automatically Evaluating Topic Coherence and Topic Model Quality Jey Han Lau Dept of Philosophy King’s College London jeyhan.lau@gmail.com David Newman Google dnewman@google.com Timothy Baldwin Dept of Computing and Information Systems The University of Melbourne tb@ldwin.net Abstract Topic models based on latent Dirichlet al- location and related methods are used in a range of user-focused tasks including doc- ument navigation and trend analysis, but evaluation of the intrinsic quality of the topic model and topics remains an open research area. In this work, we explore the two tasks of automatic evaluation of single topics and automatic evaluation of whole topic models, and provide recom- mendations on the best strategy for per- forming the two tasks, in addition to pro- viding an open-source toolkit for topic and topic model evaluation. 1 Introduction Topic modelling based on Latent Dirichlet Alloca- tion (LDA: Blei et al. (2003)) and related methods is increasingly being used in user-focused tasks, in contexts such as the evaluation of scientific impact (McCallum et al., 2006; Hall et al., 2008), trend analysis (Bolelli et al., 2009; Lau et al., 2012a) and document search (Wang et al., 2007). The LDA model is based on the assumption that doc- ument collections have latent topics, in the form of a multinomial distribution of words, which is typically presented to users via its top-N highest- probability words. In NLP, topic models are gener- ally used as a means of preprocessing a document collection, and the topics and per-document topic allocations are fed into downstream applications such as document summarisation (Haghighi and Vanderwende, 2009), novel word sense detection methods (Lau et al., 2012b) and machine transla- tion (Zhao and Xing, 2007). In fields such as the digital humanities, on the other hand, human users interact directly with the output of topic models. It is this context of topic modelling for direct human consumption that we target in this paper. The topics produced by topic models have a varying degree of human-interpretability. To il- lustrate this, we present two topics automatically learnt from a collection of news articles: 1. ⟨farmers, farm, food, rice, agriculture⟩ 2. ⟨stories, undated, receive, scheduled, clients⟩ The first topic is clearly related to agriculture. The subject of the second topic, however, is less clear, and may confuse users if presented to them as part of a larger topic model. Measuring the human-interpretability of topics and the overall topic model is the core topic of this paper. Various methodologies have been proposed for measuring the semantic interpretability of topics. In Chang et al. (2009), the authors proposed an indirect approach based on word intrusion, where “intruder words” are randomly injected into topics and human users are asked to identify the intruder words. The word intrusion task builds on the as- sumption that the intruder words are more iden- tifiable in coherent topics …

Content truncated to 500 words