Incorporating Global Information in Local Attention for Knowledge Representation Learning

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1341–1351 August 1–6, 2021. ©2021 Association for Computational Linguistics 1341 Incorporating Global Information in Local Attention for Knowledge Representation Learning Yu Zhao1,∗, Han Zhou2,*, Ruobing Xie3, Fuzhen Zhuang4,5, Qing Li1, Ji Liu6 1Fintech Innovation Center, Financial Intelligence and Financial Engineering Key Laboratory, Southwestern University of Finance and Economics, Chengdu, China 2Department of Computer Science, The University of Hong Kong, China 3WeChat Search Application Department, Tencent, Beijing, China 4 Institute of Artificial Intelligence, Beihang University, Beijing 100191, China 5Xiamen Data Intelligence Academy of ICT, CAS, China 6AI platform and Seattle AI Lab, Kwai Inc. Abstract Graph Attention Networks (GATs) have proven a promising model that takes advantage of localized attention mechanism to perform knowledge representation learning (KRL) on graph-structure data, e.g., Knowledge Graphs (KGs). While such approaches model entities’ local pairwise importance, they lack the capa- bility to model global importance relative to other entities of KGs. This causes such mod- els to miss critical information in tasks where global information is also a significant compo- nent for the task, such as in knowledge repre- sentation learning. To address the issue, we allow the proper incorporation of global infor- mation into the GAT family of models through the use of scaled entity importance, which is calculated by an attention-based global ran- dom walk algorithm. In the context of KRL, in- corporating global information boosts perfor- mance significantly. Experimental results on KG entity prediction against the state-of-the- arts sufficiently demonstrate the effectiveness of our proposed model. 1 Introduction Graph Attention Networks (GATs) have been suc- cessfully applied to various tasks over graphs (Velickovic et al., 2018; Lee et al., 2018b), such as graph classification (Wu et al., 2019b; Lee et al., 2018a), link prediction (Abu-El-Haija et al., 2018), and node classification (Lee et al., 2019; Zhang et al., 2020a). GATs learn from the underlying graph structure by making use of localized atten- tion mechanism (Wu et al., 2019a; Xu et al., 2019; Vashishth et al., 2020b), where the hidden repre- sentation of each node is computed by recursively aggregating and attending over its corresponding local neighbors’ features, and the weighting coeffi- cients are calculated inductively with self-attention ∗ Equal Contribution. Corresponding author: Y. Zhao (zhaoyu@swufe.edu.cn). strategy (Thekumparampil et al., 2018; Qian et al., 2018; Zhang et al., 2018). The original GATs per- form only on single-relational homogeneous graphs (Velickovic et al., 2018; Wang et al., 2019b). Re- cent advancements were proposed to operate on more general and prevalent multi-relational graphs (Wang et al., 2019b; Hong et al., 2020; Nathani et al., 2019; Zhang et al., 2020c), such as the repre- sentative Knowledge Graphs (KGs) which contain multiple types of entities (nodes) and relationships (edges) (Zhou et al., 2018; Han et al., 2018; Wang et al., 2019a; Zhao et al., 2020). However, these ap- proaches can only exploit localized features within the neighborhood of individual entities (Nathani et al., 2019; Busbridge et al., 2019; Zhang et al., 2020c). For some tasks, such simplified …

Content truncated to 500 words