Graph Relational Topic Model with Higher-order Graph Attention Auto-encoders

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2604–2613 August 1–6, 2021. ©2021 Association for Computational Linguistics 2604 Graph Relational Topic Model with Higher-order Graph Attention Auto-encoders Qianqian Xie Department of Computer Science, University of Manchester xieq@whu.edu.cn Jimin Huang School of Computer Science, Wuhan University huangjimin@whu.edu.cn Pan Du Department of Computer Science and Operations Research, University of Montreal pandu@iro.umontreal.ca Min Peng School of Computer Science, Wuhan University pengm@whu.edu.cn Abstract Learning low-dimensional representations of networked documents is a crucial task for doc- uments linked in network structures. Rela- tional Topic Models (RTMs) have shown their strengths in modeling both document contents and relations to discover the latent topic se- mantic representations. However, higher-order correlation structure information among doc- uments is largely ignored in these methods. Therefore, we propose a novel graph relational topic model (GRTM) for document network, to fully explore and mix neighborhood infor- mation of documents on each order, based on the Higher-order Graph Attention Network (HGAT) with the log-normal prior in the graph attention. The proposed method can ad- dress the aforementioned issue via the informa- tion propagation among document-document based on the HGAT probabilistic encoder, to learn efficient networked document representa- tions in the latent topic space, which can fully reflect document contents, along with docu- ment connections. Experiments on several real-world document network datasets show that, through fully exploring information in documents and document networks, our model achieves better performance on unsupervised representation learning and outperforms ex- isting competitive methods in various down- stream tasks. 1 Introduction Document networks, such as hyperlink networks of Web pages, citation networks of academic doc- uments, and user profiles in social networks, have long been an intensively studied research sub- ject due to their wide applications. Finding low- dimensional representations of networked docu- ments to preserve document contents and connec- tions among documents simultaneously is a cru- cial research task. Inspired by the wide applica- tion of topic models such as latent Dirichlet allo- cation (LDA) (Blei et al., 2003) on discovering the latent semantic structure of unconnected docu- ments, a series of Relational Topic Models (RTMs) are proposed to explore the latent topic seman- tic structure of documents and links among them, based on probabilistic graphical models (Nallapati et al., 2008; Chang and Blei, 2009; Le and Lauw, 2014; Chen et al., 2014; Yang et al., 2016), deep generative models (Acharya et al., 2015; Wang et al., 2017; Bai et al., 2018), auto-encoders (AEs) (Zhang and Lauw, 2020) and graph auto-encoders (GAEs) (Wang et al., 2020a). However, most RTMs consider only the pair- wise correlation or the first-order neighbor corre- lation (Zhang and Lauw, 2020) among documents. Although the recently proposed deep relational topic model, GPFA (Wang et al., 2020a) based on graph neural networks (GNNs) can consider low-order indirect neighborhood information via stacked graph neural network (GNN) (Kipf and Welling, 2016b) layers, it still suffers from exploit- ing the deep interactions (higher-order) between indirectly connected documents due to the over- smoothing problem (Li et al., 2018). …

Content truncated to 500 words