Nonparametric Forest-Structured Neural Topic Modeling

Proceedings of the 29th International Conference on Computational Linguistics, pages 2585–2597 October 12–17, 2022. 2585 Nonparametric Forest-Structured Neural Topic Modeling Zhihong Zhang∗, Xuewen Zhang∗, Yanghui Rao† School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, China {zhangzhh33, zhangxw53}@mail2.sysu.edu.cn, raoyangh@mail.sysu.edu.cn Abstract Neural topic models have been widely used in discovering the latent semantics from a cor- pus. Recently, there are several researches on hierarchical neural topic models since the re- lationships among topics are valuable for data analysis and exploration. However, the exist- ing hierarchical neural topic models are limited to generate a single topic tree. In this study, we present a nonparametric forest-structured neural topic model by firstly applying the self- attention mechanism to capture parent-child topic relationships, and then build a sparse di- rected acyclic graph to form a topic forest. Ex- periments indicate that our model can automat- ically learn a forest-structured topic hierarchy with indefinite numbers of trees and leaves, and significantly outperforms the baseline models on topic hierarchical rationality and affinity. 1 Introduction Topic model has been widely used in modeling a collection of documents and encoding the text content to a low dimensional feature space. Tradi- tional topic models can be divided into probabilistic graphical models and matrix factorization based methods. Probabilistic graphical models, such as Latent Dirichlet Allocation (LDA) (Blei et al., 2003b), rely on approximate approaches (e.g., vari- ational inference and Gibbs sampling) with com- plex derivation or high computational costs to esti- mate parameters (Srivastava and Sutton, 2017). Ma- trix factorization based methods (Lee and Seung, 1999) can effectively decompose the document- word representation into two sub-matrices but are subject to a low stability (Chen et al., 2021b). Re- cently, neural topic models based on Neural Vari- ational Inference (NVI) (Srivastava and Sutton, 2017; Miao et al., 2017; Chen et al., 2021b) have attracted great attention owing to the advantages of fast parameter inference and flexibility. ∗The first two authors contributed equally to this work. †The corresponding author. email mail write dear time support matter people involve vote president government vote support right community house agree involve home spirit god christ christian jews result study research specific role patients risk clinical disease years cell neurons nucleus nerve cells protein genes gene express sequence pain rat infect infection rats cardiac man ventricular heart risk dna rna viral replication virus Level 1 Level 2 Level 3 Figure 1: An example of forest-structured topics. Each topic is represented by 5 top words. Despite some notable successes for neural topic models, most of the existing methods can only ex- tract topics at the same level. This may cause confu- sions on identifying the hierarchical structure about the relationships among topics, which is valuable for data analysis and exploration in various do- mains (Paisley et al., 2015). To address this, a few neural topic models have been developed to model the hierarchical structure of topics (Isonuma et al., 2020; Chen et al., 2021b,a). The above methods, however, all assume that the hierarchical structure is a tree …

Content truncated to 500 words