The Sensitivity of Topic Coherence Evaluation to Topic Cardinality

Proceedings of NAACL-HLT 2016, pages 483–487, San Diego, California, June 12-17, 2016. c⃝2016 Association for Computational Linguistics The Sensitivity of Topic Coherence Evaluation to Topic Cardinality Jey Han Lau1,2 and Timothy Baldwin2 1 IBM Research 2 Dept of Computing and Information Systems, The University of Melbourne jeyhan.lau@gmail.com, tb@ldwin.net Abstract When evaluating the quality of topics gener- ated by a topic model, the convention is to score topic coherence — either manually or automatically — using the top-N topic words. This hyper-parameter N, or the cardinality of the topic, is often overlooked and selected arbitrarily. In this paper, we investigate the impact of this cardinality hyper-parameter on topic coherence evaluation. For two auto- matic topic coherence methodologies, we ob- serve that the correlation with human ratings decreases systematically as the cardinality in- creases. More interestingly, we find that per- formance can be improved if the system scores and human ratings are aggregated over several topic cardinalities before computing the cor- relation. In contrast to the standard practice of using a fixed value of N (e.g. N = 5 or N = 10), our results suggest that calculating topic coherence over several different cardi- nalities and averaging results in a substantially more stable and robust evaluation. We release the code and the datasets used in this research, for reproducibility.1 1 Introduction Latent Dirichlet Allocation (“LDA”: Blei et al. (2003)) is an approach to document clustering, in which “topics” (multinomial distributions over terms) and topic allocations (multinomial distribu- tions over topics per document) are jointly learned. When the topic model output is to be presented 1https://github.com/jhlau/ topic-coherence-sensitivity to humans, optimisation of the number of topics is a non-trivial problem. In the seminal paper of Chang et al. (2009), e.g., the authors showed that — contrary to expectations — extrinsically measured topic coherence correlates negatively with model perplexity. They introduced the word intrusion task, whereby a randomly selected “intruder” word is in- jected into the top-N words of a given topic and users are asked to identify the intruder word. Low reliability in identifying the intruder word indicates low coherence (and vice versa), based on the in- tuition that the more coherent the topic, the more clearly the intruder word should be an outlier. Since then, several methodologies have been in- troduced to automate the evaluation of topic coher- ence. Newman et al. (2010) found that aggregate pairwise PMI scores over the top-N topic words correlated well with human ratings. Mimno et al. (2011) proposed replacing PMI with conditional probability based on co-document frequency. Ale- tras and Stevenson (2013) showed that coherence can be measured by a classical distributional similar- ity approach. More recently, Lau et al. (2014) pro- posed a methodology to automate the word intrusion task directly. Their results also reveal the differences between these methodologies in their assessment of topic coherence. A hyper-parameter in all these methodologies is the number of topic words, or its cardinality. These methodologies evaluate coherence over the top-N topic words, where N is selected arbitrarily: for …

Content truncated to 500 words