Neural Topic Modeling by Incorporating Document Relationship Graph

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 3790–3796, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 3790 Neural Topic Modeling by Incorporating Document Relationship Graph Deyu Zhou∗ Xuemeng Hu Rui Wang School of Computer Science and Engineering, Key Laboratory of Computer Network and Information Integration, Ministry of Education, Southeast University, China {d.zhou,xuemenghu,rui wang}@seu.edu.cn Abstract Graph Neural Networks (GNNs) that capture the relationships between graph nodes via mes- sage passing have been a hot research direc- tion in the natural language processing com- munity. In this paper, we propose Graph Topic Model (GTM), a GNN based neural topic model that represents a corpus as a document relationship graph. Documents and words in the corpus become nodes in the graph and are connected based on document-word co- occurrences. By introducing the graph struc- ture, the relationships between documents are established through their shared words and thus the topical representation of a document is enriched by aggregating information from its neighboring nodes using graph convolution. Extensive experiments on three datasets were conducted and the results demonstrate the ef- fectiveness of the proposed approach. 1 Introduction Probabilistic topic models (Blei, 2012) are tools for discovering main themes from large corpora. The popular Latent Dirichlet Allocation (LDA) (Blei et al., 2003) and its variants (Lin and He, 2009; Zhao et al., 2010; Zhou et al., 2014) are effective in extracting coherent topics in an interpretable manner, but usually at the cost of designing so- phisticated and model-specific learning algorithm. Recently, neural topic modeling that utilizes neural- network-based black-box inference has been the main research direction in this field. Notably, NVDM (Miao et al., 2016) employs variational autoencoder (VAE) (Kingma and Welling, 2013) to model topic inference and document generation. Specifically, NVDM consists of an encoder infer- ring topics from documents and a decoder generat- ing documents from topics, where the latent topics are constrained by a Gaussian prior. Srivastava and ∗Corresponding author. Sutton (2017) argued that Dirichlet distribution is a more appropriate prior for topic modeling than Gaussian in NVDM and proposed ProdLDA that approximates the Dirichlet prior with logistic nor- mal. There are also attempts that directly enforced a Dirichlet prior on the document topics. W-LDA (Nan et al., 2019) models topics in the Wasserstein autoencoders (Tolstikhin et al., 2017) framework and achieves distribution matching by minimizing their Maximum Mean Discrepancy (MMD) (Gret- ton et al., 2012), while adversarial topic model (Wang et al., 2019a,b, 2020) directly generates doc- uments from the Dirichlet prior and such a process is adversarially trained with a discriminator under the framework of Generative Adversarial Network (GAN) (Goodfellow et al., 2014). Recently, due to the effectiveness of Graph Neu- ral Networks (GNNs) (Li et al., 2015; Kipf and Welling, 2016; Zhou et al., 2018) in embedding graph structures, there is a surge of interests of ap- plying GNN to natural language processing tasks (Yasunaga et al., 2017; Song et al., 2018; Yao et al., 2019). For example, GraphBTM (Zhu et al., 2018) is …

Content truncated to 500 words