Evaluating Topic Quality with Posterior Variability

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3471–3477, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 3471 Evaluating Topic Quality with Posterior Variability Linzi Xing† and Michael J. Paul‡ and Giuseppe Carenini† † University of British Columbia, Vancouver, Canada ‡ University of Colorado, Boulder, Colorado, USA {lzxing, carenini}@cs.ubc.ca, mpaul@colorado.edu Abstract Probabilistic topic models such as latent Dirichlet allocation (LDA) are popularly used with Bayesian inference methods such as Gibbs sampling to learn posterior distributions over topic model parameters. We derive a novel measure of LDA topic quality using the variability of the posterior distributions. Com- pared to several existing baselines for auto- matic topic evaluation, the proposed metric achieves state-of-the-art correlations with hu- man judgments of topic quality in experiments on three corpora.1 We additionally demon- strate that topic quality estimation can be fur- ther improved using a supervised estimator that combines multiple metrics. 1 Introduction Latent Dirichlet Allocation (LDA) (Blei et al., 2003) topic modeling has been widely used for NLP tasks which require the extraction of latent themes, such as scientific article topic analysis (Hall et al., 2008), news media tracking (Roberts et al., 2013), online campaign detection (Paul and Dredze, 2014) and medical issue analysis (Huang et al., 2015, 2017). To reliably utilize topic models trained for these tasks, we need to evaluate them carefully and ensure that they have as high qual- ity as possible. When topic models are used in an extrinsic task, like text categorization, they can be assessed by measuring how effectively they con- tribute to that task (Chen et al., 2016; Huang et al., 2015). However, when they are generated for hu- man consumption, their evaluation is more chal- lenging. In such cases, interpretability is critical, and Chang et al. (2009); Aletras and Stevenson (2013) have shown that the standard way to eval- uate the output of a probabilistic model, by mea- suring perplexity on held-out data (Wallach et al., 1Our code and data are available here. 2009), does not imply that the inferred topics are human-interpretable. A topic inferred by LDA is typically represented by the set of words with the highest probability given the topic. With this characteristic, we can evaluate the topic quality by determining how co- herent the set of topic words is. While a variety of techniques (Section 2) have been geared towards measuring the topic quality in this way, in this pa- per, we push such research one step further by making the following two contributions: (1) We propose a novel topic quality metric by using the variability of LDA posterior distributions. This metric conforms well with human judgments and achieves the state-of-the-art performance. (2) We also create a topic quality estimator by combining two complementary classes of metrics: the metrics that use information from posterior distributions (including our new metric), along with the set of metrics that rely on topic word co-occurrence. Our novel estimator further …

Content truncated to 500 words