Nonlinear Structural Equation Model Guided Gaussian Mixture Hierarchical Topic Modeling

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 10377–10390 July 9-14, 2023 ©2023 Association for Computational Linguistics Nonlinear Structural Equation Model Guided Gaussian Mixture Hierarchical Topic Modeling Hegang Chen and Pengbo Mao and Yuyin Lu and Yanghui Rao∗ School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, China {chenhg25,maopb,luyy37}@mail2.sysu.edu.cn, raoyangh@mail.sysu.edu.cn Abstract Hierarchical topic models, which can extract semantically meaningful topics from a text corpus in an unsupervised manner and auto- matically organise them into a topic hierar- chy, have been widely used to discover the underlying semantic structure of documents. However, the existing models often assume in the prior that the topic hierarchy is a tree structure, ignoring symmetrical dependencies between topics at the same level. Moreover, the sparsity of text data often complicate the analysis. To address these issues, we propose NSEM-GMHTM as a deep topic model, with a Gaussian mixture prior distribution to im- prove the model’s ability to adapt to sparse data, which explicitly models hierarchical and symmetric relations between topics through the dependency matrices and nonlinear struc- tural equations. Experiments on widely used datasets show that our NSEM-GMHTM gener- ates more coherent topics and a more rational topic structure when compared to state-of-the- art baselines. Our code is available at https: //github.com/nbnbhwyy/NSEM-GMHTM. 1 Introduction Topic models, which can uncover the hidden se- mantic structure in a text corpus, have been widely applied to text analysis. Specifically, a topic model aims to discover a set of semantically meaningful topics from a document set. Each topic captures a common pattern of word co-occurrences in the document and is often interpreted semantically as a coherent set of words representing a common concept. Although traditional topic models like the Latent Dirichlet Allocation (LDA) (Blei et al., 2003b) and the Embedded Topic Model (ETM) (Dieng et al., 2020) are able to achieve this goal, they assume that topics are independent, which lim- its the ability of these models to explore the topic ∗The corresponding author. a b Topic nodes at different levels Figure 1: Illustration of (a) topics discovered by previous hier- archical topic models, and (b) topics found by our hierarchical topic model guided by nonlinear structural equations. structure. To remedy the defect, a series of hier- archical extensions, such as the hierarchical LDA (hLDA) (Blei et al., 2003a), the recursive Chinese Restaurant Process (rCRP) (Kim et al., 2012), and the nested Hierarchical Dirichlet Process (nHDP) (Paisley et al., 2015), have been proposed. Com- monly, these models learn hierarchical topics in tree structures, which assume that the topics in the upper layers are more general/abstract than those in the lower layers. Consequently, reveal- ing hierarchical relations between topics provides the user an intuitive way to better understand text data. However, these methods rely on approximate approaches (e.g., variational inference and Gibbs sampling) and require complex derivation or high computational costs to estimate parameters. With the development of deep neural networks and the proposal of Neural Variational Inference (NVI), there is …

Content truncated to 500 words