Tree-Structured Topic Modeling with Nonparametric Neural Variational Inference

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 2343–2353 August 1–6, 2021. ©2021 Association for Computational Linguistics 2343 Tree-Structured Topic Modeling with Nonparametric Neural Variational Inference Ziye Chen1, Cheng Ding1, Zusheng Zhang1, Yanghui Rao1,∗, Haoran Xie2 1School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, China 2Department of Computing and Decision Sciences, Lingnan University, Hong Kong {chenzy35,dingch6,zhangzsh3}@mail2.sysu.edu.cn, raoyangh@mail.sysu.edu.cn, hrxie2@gmail.com Abstract Topic modeling has been widely used for dis- covering the latent semantic structure of doc- uments, but most existing methods learn top- ics with a flat structure. Although probabilis- tic models can generate topic hierarchies by introducing nonparametric priors like Chinese restaurant process, such methods have data scalability issues. In this study, we develop a tree-structured topic model by leveraging non- parametric neural variational inference. Par- ticularly, the latent components of the stick- breaking process are first learned for each doc- ument, then the affiliations of latent compo- nents are modeled by the dependency matri- ces between network layers. Utilizing this network structure, we can efficiently extract a tree-structured topic hierarchy with reason- able structure, low redundancy, and adaptable widths. Experiments on real-world datasets validate the effectiveness of our method. 1 Introduction Topic models (Blei et al., 2003; Griffiths et al., 2004) are important tools for discovering latent se- mantic patterns in a corpus. These models can be grouped into flat models and hierarchical models. In many domains, topics can be naturally orga- nized into a tree, where the hierarchical relation- ships among topics are valuable for data analy- sis and exploration. Tree-structured topic model (Griffiths et al., 2004) was thus developed to learn coherent topics from text without disrupting the inherent hierarchical structure. Such a method has been proven as useful in various downstream ap- plications, including hierarchical categorization of Web pages (Ming et al., 2010), aspects hierarchies extraction in reviews (Kim et al., 2013), and hier- archies discovery of research topics in academic repositories (Paisley et al., 2014). ∗The corresponding author. Despite the practical importance and potential advantages, tree-structured topic models still face the following challenges. Firstly, the hierarchical structure of topics should be reasonable (Viegas et al., 2020). Typically, topics near the root are more general while the ones close to the leaves are more specific. Besides, child topics should be coherent with their corresponding parent top- ics. Secondly, low redundancy is necessary for the extracted topics, in order to prevent the dis- tributions associated with parent topics and their children being extremely similar (Griffiths et al., 2004). Thirdly, the number of topics in each hier- archy level should be automatically determined by the model, because it is usually unknown and can not be previously set to a predefined value (Kim et al., 2012). Finally, it is difficult for probabilistic models to enhance the data scalability (Isonuma et al., 2020). Previously, several tree-structured topic models (Griffiths et al., 2004; Kim et al., 2012; Isonuma et al., 2020) have been developed. But …

Content truncated to 500 words