Disentangle-based Continual Graph Representation Learning

Disentangle-based Continual Graph Representation Learning Xiaoyu Kou1∗, Yankai Lin2, Shaobo Liu2, Peng Li2, Jie Zhou2, Yan Zhang1 1Key Laboratory of Machine Perception (MOE), Department of Machine Intelligence, Peking University, Beijing, China 2Pattern Recognition Center, WeChat AI, Tencent Inc., China {kouxiaoyu,zhyzhy001}@pku.edu.cn {yankailin,patrickpli,withtomzhou}@tencent.com Abstract Graph embedding (GE) methods embed nodes (and/or edges) in graph into a low-dimensional semantic space, and have shown its effective- ness in modeling multi-relational data. How- ever, existing GE models are not practical in real-world applications since it overlooked the streaming nature of incoming data. To ad- dress this issue, we study the problem of con- tinual graph representation learning which aims to continually train a graph embedding model on new data to learn incessantly emerg- ing multi-relational data while avoiding catas- trophically forgetting old learned knowledge. Moreover, we propose a disentangle-based continual graph representation learning (DiC- GRL) framework inspired by the human’s abil- ity to learn procedural knowledge. The exper- imental results show that DiCGRL could ef- fectively alleviate the catastrophic forgetting problem and outperform state-of-the-art con- tinual learning models. 1 Introduction Multi-relational data represents relationships be- tween entities in the world, which is usually de- noted as a multi-relational graph with nodes and edges connecting them. It is widely used in real- world NLP applications such as knowledge graphs (KGs) (e.g., Freebase (Bollacker et al., 2008) and DBpedia (Lehmann et al., 2015)) and informa- tion networks (e.g., social network and citation network). Therefore, modeling multi-relational graph with graph embeddings (Bordes et al., 2013; Tang et al., 2015a; Sun et al., 2019; Bruna et al., 2014) has been attracting intensive attentions in both academia and industry. Graph embedding (GE), aiming to embed nodes and/or edges in the graph into a low-dimensional semantic space to ∗This work is done when Xiaoyu Kou was interning at Pattern Recognition Center, WeChat AI, Tencent Inc, China !"#"$% &'"(" )*$+,--, &'"(" )"-*" .// &'"(" .//, 01/+"( 2#,3*4,/5 6+, 7/*5,4 85"5,3 85"5, 9: ;"<"** ������ ="<>,# ?#"3,# @. B9'*/39/ ?*#35 ="4> 9: 78 ������ ������������������ ������ �������� �������� ������ ��������� ������������ @+*$"C9 ��������� ���������� Figure 1: An example of edges that compressed into multiple components, which represented in different colors. The dotted line represents the inferred proce- dural knowledge from two bold edges. enable neural models to effectively and efficiently utilize multi-relational data, has demonstrated re- markable effectiveness in various downstream NLP tasks such as question answering (Bordes et al., 2014) and dialogue system (Moon et al., 2019). Nevertheless, most existing graph embedding works overlook the streaming nature of the incom- ing data in real-world scenarios. In consequence, these models have to be retrained from scratch to reflect the data change, which is computationally expensive. To tackle this issue, we propose to study the problem of continual graph represen- tation learning (CGRL) in this work. The goal of continual learning is to alleviate catastrophically forgetting old data while learn- ing new data. There are two mainstream contin- ual learning methods in NLP: (1) consolidation- based methods (Kirkpatrick et al., 2017; Zenke et al., …

Content truncated to 500 words