Apples to Apples: A Systematic Evaluation of Topic Models

Proceedings of Recent Advances in Natural Language Processing, pages 483–493 Sep 1–3, 2021. https://doi.org/10.26615/978-954-452-072-4_055 483 Apples to Apples: A Systematic Evaluation of Topic Models Ismail Harrando, Pasquale Lisena and Rapha¨el Troncy EURECOM, Sophia Antipolis, France {firstname.lastname}@eurecom.fr Abstract From statistical to neural models, a wide va- riety of topic modelling algorithms have been proposed in the literature. However, because of the diversity of datasets and metrics, there have not been many efforts to systematically compare their performance on the same bench- marks and under the same conditions. In this paper, we present a selection of 9 topic mod- elling techniques from the state of the art re- flecting a diversity of approaches to the task, an overview of the different metrics used to compare their performance, and the challenges of conducting such a comparison. We empiri- cally evaluate the performance of these mod- els on different settings reflecting a variety of real-life conditions in terms of dataset size, number of topics, and distribution of topics, following identical preprocessing and evalua- tion processes. Using both metrics that rely on the intrinsic characteristics of the dataset (different coherence metrics), as well as exter- nal knowledge (word embeddings and ground- truth topic labels), our experiments reveal sev- eral shortcomings regarding the common prac- tices in topic models evaluation. 1 Introduction The automatic analysis of textual data has gained increasing levels of attention over the last few decades. The cost of manually analysing and anno- tating the ever-growing quantity of content created and shared on the Web continues to be prohibitively expensive. Topic modelling is an NLP task where, given a corpus of documents, the objective is to find the underlying meaningful clusters of docu- ments (or topics) that are thematically coherent (use consistent and related vocabulary) and assign each document to one or more of these topics. As a text mining technique, it allows the analysis of big volumes of textual documents through clus- tering them into coherent sets addressing similar subjects (or topics), and labeling them using key- words that are understandable by the end-user. It has the advantage of not relying on any labeled data to achieve good results, as the training of topic models is done in an unsupervised matter. More- over, the resulting topics and representations can then be used to perform other NLP tasks such as trend prediction (Lau et al., 2012), text summariza- tion (Lin and Hovy, 2000), improving named entity recognition (Newman et al., 2006), and content recommendation (Papneja et al., 2021). Because of the unsupervised nature of the task, the evaluation of the quality of topic modelling techniques relies usually on metrics that do not require human annotation or ground-truth labels. Most of the used “coherence” metrics – further detailed in Section 3.1 – attempt to measure how much the resulting topics reflect some statistical characteristics of the original dataset and its word co-occurrences distribution. These metrics utilise different definitions of what a “coherent topic” is, and they only contingently agree with humans judgement (Chang et …

Content truncated to 500 words