Evaluating Unsupervised Hierarchical Topic Models Using a Labeled Dataset

Proceedings of Recent Advances in Natural Language Processing, pages 846–853 Varna, Sep 4–6, 2023 https://doi.org/10.26615/978-954-452-092-2_091 846 Evaluating Unsupervised Hierarchical Topic Models Using a Labeled Dataset Judicael Poumay ULiege/HEC Liege Rue Louvrex 14, 4000 Liege, Belgium judicael.poumay@uliege.be Ashwin Ittoo ULiege/HEC Liege Rue louvrex 14, 4000 Liege, Belgium ashwin.ittoo@uliege.be Abstract Topic models are often evaluated with mea- sures such as perplexity and topic coherence. However, these methods fall short in determin- ing the comprehensiveness of identified topics. This research introduces a complementary ap- proach to evaluating unsupervised topic models using a labeled dataset. By training hierarchi- cal topic models and utilizing known labels for evaluation, we found a high accuracy of 70% for expected topics. Despite having 90 labels in the dataset, even those representing only 1% of the data achieved an average accuracy of 37.9%, illustrating hierarchical topic models’ effectiveness on smaller subsets. Additionally, we confirmed that this new evaluation method helps assess the topic tree quality, demonstrat- ing that hierarchical topic models generate co- herent taxonomies. Lastly, we established that coherence measures alone are insufficient for a holistic topic model evaluation. 1 Introduction Hierarchical Topic Models such as the LSHTM(Pujara and Skomoroch, 2012), nCRP(Blei et al., 2004), nHDP(Paisley et al., 2015), and HTMOT(Poumay and Ittoo, 2021) enable the extraction of topics and sub-topics organized in a tree-like hierarchy. Topic hierarchies provide a more fine-grained view of the underlying data, which is particularly useful in applications such as ontology learning (Zhu et al., 2017) and research idea recommendation(Wang et al., 2019). Additionally, models like nCRP, NHDP, and HTMOT dynamically determine the appropriate number of topics and sub-topics during training, contrary to the traditional model of LDA(Blei et al., 2003). Evaluating the quality of the extracted topics is crucial to ascertain their real-world utility. How- ever, as these methods extract knowledge in an unsupervised manner, previous studies on topic model evaluation have been limited to evaluating the quality of the resulting topics. Hence, many methods have been proposed to study the perfor- mance of these models, such as perplexity and co- herence measures (Newman et al., 2010; Doogan and Buntine, 2021a; Bhatia et al., 2017). Nevertheless, these measures have proven to be unrelated to human judgment (Chang et al., 2009; Doogan and Buntine, 2021b; Bhatia et al., 2017), indicating that humans do not agree with these measures when it comes to the quality of the top- ics extracted. Recently, the word intrusion task has been proposed to evaluate the extracted topic quality (Chang et al., 2009). While its initial im- plementation relies on human annotators, it can be automated without losing the link to human judg- ment (Lau et al., 2014). However, all the methods previously presented have failed to ask other essential questions about the extracted topics and the completeness of the results. For example: Do we extract every topic? How well do we extract them? Do we extract unex- pected topics? And in the context of hierarchical topic models, is the hierarchy produced coherent? Hence, in this article, we propose …

Content truncated to 500 words