Topic Modeling With Topological Data Analysis

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11514 - 11533 December 7-11, 2022 ©2022 Association for Computational Linguistics Topic Modelling with Topological Data Analysis Ciarán Byrne ∗ ciaranby@gmail.com Danijela Horak ∗ danijela.horak@gmail.com Amandla Mabona ∗ akvmabona@gmail.com Karo Moilanen ∗ karohenrim@gmail.com Abstract Recent unsupervised topic modelling ap- proaches that use clustering techniques on word, token or document embeddings can ex- tract coherent topics. A common limitation of such approaches is that they reveal noth- ing about inter-topic relationships which are essential in many real-world application do- mains. We present an unsupervised topic mod- elling method which harnesses Topological Data Analysis (TDA) to extract a topological skeleton of the manifold upon which contextu- alised word embeddings lie. We demonstrate that our approach, which performs on par with a recent baseline, is able to construct a network of coherent topics together with meaningful relationships between them. 1 Introduction Unsupervised topic modelling is a standard tech- nique for making sense of document collections. While traditional approaches such as LDA (Blei et al., 2003) rely on probabilistic models, the field has recently moved towards clustering-based meth- ods in which topic clusters are obtained via docu- ment, word or token embeddings (Thompson and Mimno, 2020; Silburt et al., 2021; Angelov, 2020; Grootendorst, 2022). Even though clustering can yield interpretable topics, it typically discards infor- mation about relationships between clusters, hence making it harder to interpret clusters in global con- texts. In this work, we approach topic modelling as a task to find regions on a manifold of contextu- alised word embeddings which reflect a "topic". To this end, we apply Mapper - an algorithm from the field of Topological Data Analysis (TDA). Map- per creates a graph whose topology reflects the shape of the underlying data set and whose nodes represent subsets of data points. In the case of ∗Work carried out at AIG (American International Group, Inc.) contextualised word embeddings, we construct a graph where each node is a cluster of tokens (i.e. a "topic"), and where connections between them reflect the topology of the embedding manifold. We use community detection techniques to demon- strate that semantically related topics are connected in the graph. Our main contributions are the following: 1. We propose and evaluate a new method for topic modelling which learns topics and rela- tionships between them without any restric- tions on graph structure. To the best of our knowledge, our work is the first application of TDA Mapper to the task of topic modelling. 2. To the best of our knowledge, we are the first to use stability analysis for Mapper on a real- world data set and problem. Unlike prior ap- proaches which are computationally infeasi- ble on large data sets, we propose a scalable approach using separate stability scores for both the graph topology and the clustering. 3. We define a new stability score via spectral distance between Mapper graphs. 4. We use community detection techniques to automatically identify regions of interest in large …

Content truncated to 500 words