Improving Textual Network Embedding with Global Attention via Optimal Transport

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5193–5202 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 5193 Improving Textual Network Embedding with Global Attention via Optimal Transport Liqun Chen, Guoyin Wang, Chenyang Tao, Dinghan Shen, Pengyu Cheng, Xinyuan Zhang, Wenlin Wang, Yizhe Zhang, Lawrence Carin liqun.chen@duke.edu Abstract Constituting highly informative network em- beddings is an important tool for network analysis. It encodes network topology, along with other useful side information, into low- dimensional node-based feature representa- tions that can be exploited by statistical mod- eling. This work focuses on learning context- aware network embeddings augmented with text data. We reformulate the network- embedding problem, and present two novel strategies to improve over traditional atten- tion mechanisms: (i) a content-aware sparse attention module based on optimal transport, and (ii) a high-level attention parsing mod- ule. Our approach yields naturally sparse and self-normalized relational inference. can capture long-term interactions betweenIt sequences, thus addressing the challenges faced by existing textual network embedding schemes. Extensive experiments are con- ducted to demonstrate our model can consis- tently outperform alternative state-of-the-art methods. 1 Introduction When performing network embedding, one maps network nodes into vector representations that re- side in a low-dimensional latent space. Such tech- niques seek to encode topological information of the network into the embedding, such as affin- ity (Tang and Liu, 2009), local interactions (e.g, local neighborhoods) (Perozzi et al., 2014), and high-level properties such as community struc- ture (Wang et al., 2017). Relative to classical network-representation learning schemes (Zhang et al., 2018a), network embeddings provide a more fine-grained representation that can be easily re- purposed for other downstream applications (e.g., node classification, link prediction, content rec- ommendation and anomaly detection). For real-world networks, one naturally may have access to rich side information about each node. Of particular interest are textual networks, where the side information comes in the form of natural language sequences (Le and Lauw, 2014). For example, user profiles or their online posts on social networks (e.g., Facebook, Twitter), and doc- uments in citation networks (e.g., Cora, arXiv). The integration of text information promises to significantly improve embeddings derived solely from the noisy, sparse edge representations (Yang et al., 2015). Recent work has started to explore the joint em- bedding of network nodes and the associated text for abstracting more informative representations. Yang et al. (2015) reformulated DeepWalk embed- ding as a matrix factorization problem, and fused text-embedding into the solution, while Sun et al. (2016) augmented the network with documents as auxiliary nodes. Apart from direct embedding of the text content, one can first model the topics of the associated text (Blei et al., 2003) and then sup- ply the predicted labels to facilitate embedding (Tu et al., 2016). Many important downstream applications of network embeddings are context-dependent, since a static vector representation of the nodes adapts to the changing context less effectively (Tu et al., 2017). For example, the interactions between social network users are context-dependent (e.g., …

Content truncated to 500 words