Large-Scale Correlation Analysis of Automated Metrics for Topic Models

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 13874–13898 July 9-14, 2023 ©2023 Association for Computational Linguistics Large-Scale Correlation Analysis of Automated Metrics for Topic Models Jia Peng Lim Singapore Management University jiapeng.lim.2021@smu.edu.sg Hady W. Lauw Singapore Management University hadywlauw@smu.edu.sg Abstract Automated coherence metrics constitute an im- portant and popular way to evaluate topic mod- els. Previous works present a mixed picture of their presumed correlation with human judge- ment. In this paper, we conduct a large-scale correlation analysis of coherence metrics. We propose a novel sampling approach to mine topics for the purpose of metric evaluation, and conduct the analysis via three large corpora showing that certain automated coherence met- rics are correlated. Moreover, we extend the analysis to measure topical differences between corpora. Lastly, we examine the reliability of human judgement by conducting an extensive user study, which is designed as an amalgama- tion of different proxy tasks to derive a finer insight into the human decision-making pro- cesses. Our findings reveal some correlation between automated coherence metrics and hu- man judgement, especially for generic corpora. 1 Introduction Topic modelling is an important tool in the anal- ysis and exploration of text corpora in terms of their salient topics (Blei et al., 2003). To evaluate the effectiveness of topic models, the preponder- ance of topic modeling literature rely on automated coherence metrics. A key benefit is convenience, al- lowing researchers to sidestep expensive and time- consuming user studies. The basis for this reliance is the assumption that the coherence metrics corre- late with human judgement (Mimno et al., 2011; Lau et al., 2014; Röder et al., 2015). The presumed correlation with human judge- ment should not be taken for granted. There are re- cent works that challenge the assumption. Doogan and Buntine (2021) highlight the inconsistencies of automated coherence metrics via correlation anal- ysis within each metric. In Hoyle et al. (2021), they claimed some disagreement between human judgement and automated coherence metrics. We postulate that the reasons behind such a mixed picture could be the differences in the topic samples as well as the underlying corpora from which the statistics were derived, resulting in lo- calised “biases” that affect the conclusions reached by respective studies. Given their importance, we seek to conduct an extended analysis of automated coherence metrics on a larger scale than anything previously attempted. This study includes orders of magnitudes greater than the number of topics typically analysed, covering three large corpora, employing a comprehensive user study with exten- sive labels, across most of the widely used metrics. There is a strong motivation for quantity. Given a vocabulary, a combinatorially large number of possible topics exist. If each topic is a vector of its scores on different metrics, the resulting curse of dimensionality (Bellman and Kalaba, 1959) neces- sitates a larger sample size. We argue that evaluat- ing thousands of topics might not be sufficient, and a larger sample size is required to approximate a …

Content truncated to 500 words