DIVINE: A Generative Adversarial Imitation Learning Framework for Knowledge Graph Reasoning

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 2642–2651, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 2642 DIVINE: A Generative Adversarial Imitation Learning Framework for Knowledge Graph Reasoning Ruiping Li and Xiang Cheng∗ State Key Laboratory of Networking and Switching Technology Beijing University of Posts and Telecommunications, Beijing 100876, China {liruiping,chengxiang}@bupt.edu.cn Abstract Knowledge graphs (KGs) often suffer from sparseness and incompleteness. Knowledge graph reasoning provides a feasible way to ad- dress such problems. Recent studies on knowl- edge graph reasoning have shown that rein- forcement learning (RL) based methods can provide state-of-the-art performance. Howev- er, existing RL-based methods require numer- ous trials for path-finding and rely heavily on meticulous reward engineering to fit specific dataset, which is inefficient and laborious to apply to fast-evolving KGs. In this paper, we present DIVINE, a novel plug-and-play frame- work based on generative adversarial imita- tion learning for enhancing existing RL-based methods. DIVINE guides the path-finding process, and learns reasoning policies and re- ward functions self-adaptively through imitat- ing the demonstrations automatically sampled from KGs. Experimental results on two bench- mark datasets show that our framework im- proves the performance of existing RL-based methods without extra reward engineering. 1 Introduction Knowledge graphs (Suchanek et al., 2007; Auer et al., 2007; Bollacker et al., 2008; Carlson et al., 2010; Vrandeˇci´c and Kr¨otzsch, 2014), typically composed of massive relational triples, are useful resources for many downstream natural language processing applications such as information ex- traction and question answering. Although exist- ing KGs have an extraordinarily large scale, they are still highly incomplete (Min et al., 2013), lead- ing extensive research efforts on automated infer- ence of missing information from observed evi- dence. In this paper, we focus on the problem of multi-hop reasoning in KGs, which learns explicit inference formulas from existing triples to com- plete missing ones. ∗Corresponding author To tackle multi-hop reasoning, various path- based methods (Lao et al., 2011; Gardner et al., 2013, 2014; Guu et al., 2015; Neelakantan et al., 2015; Toutanova et al., 2016; Das et al., 2017) have been proposed, which leverage the elaborate- ly selected relational paths in KGs as the reason- ing evidence. However, such evidential paths are obtained by random walks, which inevitably intro- duces inferior or even noisy paths. To address this problem, the RL-based methods, such as Deep- Path (Xiong et al., 2017) and MINERVA (Das et al., 2018), strive for more reliable evidential paths by policy-conditioned walking and achieve state-of-the-art performance. They formulate the path-finding problem as a Markov decision pro- cess where their policy-based agents continuously choose the most promising relation for state transi- tion based on the current state and reasoning pol- icy. Once a relational path found, the reasoning policy is updated by a reward function according to the path quality. Finally, through such a trial- and-error process, the well-trained policy-based a- gent can be used to find evidential paths for pre- …

Content truncated to 500 words