Automatic Evaluation of Local Topic Quality

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 788–796 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 788 Automatic and Human Evaluation of Local Topic Quality Jeffrey Lund, Piper Armstrong, Wilson Fearn, Stephen Cowley, Courtni Byun Computer Science Brigham Young University {jefflund, piper.armstrong, wfearn, scowley4, courtni.byun} @byu.edu Jordan Boyd-Graber Computer Science, iSchool, LSC, and UMIACS University of Maryland jbg@umiacs.umd.edu Kevin Seppi Computer Science Brigham Young University kseppi@byu.edu Abstract Topic models are typically evaluated with re- spect to the global topic distributions that they generate, using metrics such as coherence, but without regard to local (token-level) topic as- signments. Token-level assignments are im- portant for downstream tasks such as classi- fication. Recent models, which claim to im- prove token-level topic assignments, are only validated on global metrics. We elicit human judgments of token-level topic assignments: over a variety of topic model types and param- eters, global metrics agree poorly with human assignments. Since human evaluation is ex- pensive we propose automated metrics to eval- uate topic models at a local level. Finally, we correlate our proposed metrics with human judgments: an evaluation based on the percent of topic switches correlates most strongly with human judgment of local topic quality. This new metric, which we call consistency, should be adopted alongside global metrics such as topic coherence. 1 Introduction Topic models such as Latent Dirichlet Alloca- tion (Blei et al., 2003, LDA) automatically discover topics in a collection of documents, giving users a glimpse into themes present in the documents. LDA jointly derives a set of topics (a distribution over words) and token-topic assignments (a distri- bution over the topics for each token). While the topics by themselves are valuable, the token-topic assignments are also useful as features for docu- ment classification (Ramage et al., 2009; Nguyen et al., 2015; Lund et al., 2018) and, in principle, for topic-based document segmentation. Given the breadth of topic model variants and implementations, the question of algorithm selec- tion and model evaluation can be as daunting as it is important. When the model is used for a downstream evaluation task (e.g., document clas- sification), these questions can often be answered by maximizing downstream task performance. In other cases, automated metrics such as topic co- herence (Newman et al., 2010) can help assess topic model quality. Generally speaking, these metrics evaluate topic models globally, meaning that the metrics evaluate characteristics of the top- ics (word distributions) themselves, ignoring the topic assignments of individual tokens. In the context of human interaction, this means that models produce global topic-word distribu- tions that typically make sense to users and serve to give a good high-level overview of the general themes and trends in the data. However, the lo- cal topic assignments can be bewildering. For ex- ample, Figure 1 shows typical topic assignments using LDA. Arguably, most, if not all, of the sen- tence should be assigned to the Music topic—the sentence is about a music video for a …

Content truncated to 500 words