A User-Centered, Interactive, Human-in-the-Loop Topic Modelling System

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 505–522 May 2-6, 2023 ©2023 Association for Computational Linguistics A User-Centered, Interactive, Human-in-the-Loop Topic Modelling System Zheng Fang1, Lama Alqazlan1, Du Liu2, Yulan He1,3,4 and Rob Procter1,4 1Department of Computer Science, University of Warwick, UK 2Bayes Business School, City, University of London, UK 3Department of Informatics, King’s College London, UK 4The Alan Turing Institute, UK {Z.Fang.4, Lama.Alqazlan, Rob.Procter}@warwick.ac.uk yulan.he@kcl.ac.uk, Du.Liu@city.ac.uk Abstract Human-in-the-loop topic modelling incorpo- rates users’ knowledge into the modelling pro- cess, enabling them to refine the model itera- tively. Recent research has demonstrated the value of user feedback, but there are still issues to consider, such as the difficulty in tracking changes, comparing different models and the lack of evaluation based on real-world exam- ples of use. We developed a novel, interactive human-in-the-loop topic modeling system with a user-friendly interface that enables users com- pare and record every step they take, and a novel topic words suggestion feature to help users provide feedback that is faithful to the ground truth. Our system also supports not only what traditional topic models can do, i.e., learning the topics from the whole corpus, but also targeted topic modelling, i.e., learning top- ics for specific aspects of the corpus. In this article, we provide an overview of the system and present the results of a series of user stud- ies designed to assess the value of the system in progressively more realistic applications of topic modelling. 1 Introduction Huge amounts of unstructured, textual data are generated daily. As more data becomes available, it becomes more difficult to search, understand and discover the knowledge within it. Because of the human effort it requires, conventional qualita- tive approaches, such as Grounded Theory, (Glaser et al., 1968) are no longer feasible with such large volumes of data. Topic modelling is a potential solution that has received increasing attention in recent research (Heidenreich et al., 2019; Curiskis et al., 2020; Dantu et al., 2021; Goyal and Howlett, 2021) to help users organize, search, and under- stand large amounts of information. It is an unsu- pervised machine learning technique for identify- ing hidden topics in large, unstructured text cor- pora, in which a hidden topic is represented by a group of words describing a common theme. Users can easily identify the topics in each document and search for documents closely associated with a specific topic for a more in-depth study. However, the topics generated by conventional topic models are often incoherent and contain many unrelated words (Chang et al., 2009; Mimno et al., 2011; Boyd-Graber et al., 2014). Although these issues can be addressed by pre-processing the target data source, for example, by removing irrelevant words from the vocabulary list and adjusting the hyper- parameters of the model, such as the number of topics, this requires familiarity with the algorithm. Hence, it is difficult for anyone who does not have some knowledge of topic modelling. Human-in-the-loop topic modelling (HL-TM) incorporates human …

Content truncated to 500 words