Topic Model or Topic Twaddle? Re-evaluating Semantic Interpretability Measures

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3824–3848 June 6–11, 2021. ©2021 Association for Computational Linguistics 3824 Topic Model or Topic Twaddle? Re-evaluating Semantic Interpretability Measures Caitlin Doogan Faculty of Information Technology Monash University, Australia caitlin.doogan@dmonash.edu Wray Buntine Faculty of Information Technology Monash University, Australia wray.buntine@dmonash.edu Abstract When developing topic models, a critical ques- tion that should be asked is: How well will this model work in an applied setting? Because standard performance evaluation of topic inter- pretability uses automated measures modeled on human evaluation tests that are dissimilar to applied usage, these models’ generalizabil- ity remains in question. In this paper, we probe the issue of validity in topic model evaluation and assess how informative coherence mea- sures are for specialized collections used in an applied setting. Informed by the literature, we propose four understandings of interpretability. We evaluate these using a novel experimental framework reflective of varied applied settings, including human evaluations using open label- ing, typical of applied research. These evalu- ations show that for some specialized collec- tions, standard coherence measures may not inform the most appropriate topic model or the optimal number of topics, and current in- terpretability performance validation methods are challenged as a means to confirm model quality in the absence of ground truth data. 1 Introduction Topic modeling has become a popular tool for ap- plied research such as social media analysis, as it facilitates the exploration of large document- collections and yields insights that would not be accessible by manual methods (Sinnenberg et al., 2017; Karami et al., 2020). However, social media data can be challenging to model as it is both sparse and noisy (Zhao et al., 2011). This has resulted in increased demand for short-text topic models that can handle these challenges (Lim et al., 2013; Zuo et al., 2016; Chen et al., 2015). Topic word-sets, denoted Tws, are considered to be semantically related words that represent the la- tent component of the underlying topic’s document- collection, denoted Tdc. Meaning is derived from these topics through the interpretation of either the Tws (Nerghes and Lee, 2019), the corresponding Tdc (Maier et al., 2018), or both (Törnberg and Törnberg, 2016). Since meaning requires topics to be interpretable to humans, empirical assurance is needed to confirm a novel topic models’ capac- ity to generate “semantically interpretable” topics, as well as a method to guide model selection and other parameters such as the number of topics, K. This is often achieved by calculating the coherence scores for Tws (Lau and Baldwin, 2016) Recent literature contradicts previous evalua- tions of some short-text topic models that claim su- perior interpretability (Li et al., 2018; Eickhoff and Wieneke, 2018; Bhatia et al., 2017). Such rethink- ing flows from the fact there is no agreement on the best measure of interpretability (Lau et al., 2014b; Morstatter and Liu, 2017) and is compounded by the unclear relationship between human evaluation methodologies and automated coherence …

Content truncated to 500 words