Rule-Aware Reinforcement Learning for Knowledge Graph Reasoning

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4687–4692 August 1–6, 2021. ©2021 Association for Computational Linguistics 4687 Rule-Aware Reinforcement Learning for Knowledge Graph Reasoning Zhongni Hou1,2, Xiaolong Jin1,2, Zixuan Li1,2, Long Bai1,2 1School of Computer Science and Technology, University of Chinese Academy of Sciences; 2CAS Key Laboratory of Network Data Science and Technology, Institute of Computing Technology, Chinese Academy of Sciences {houzhongni18z,jinxiaolong,lizixuan,bailong18b}@ict.ac.cn Abstract Multi-hop reasoning is an effective and ex- plainable approach to predicting missing facts in Knowledge Graphs (KGs). It usually adopts the Reinforcement Learning (RL) framework and searches over the KG to find an evidential path. However, due to the large exploration space, the RL-based model struggles with the serious sparse reward problem and needs to make a lot of trials. Moreover, its exploration can be biased towards spurious paths that coin- cidentally lead to correct answers. To solve both problems, we propose a simple but ef- fective RL-based method called RARL (Rule- Aware RL). It injects high quality symbolic rules into the model’s reasoning process and employs partially random beam search, which can not only increase the probability of paths getting rewards, but also alleviate the impact of spurious paths. Experimental results show that it outperforms existing multi-hop methods in terms of Hit@1 and MRR. 1 Introduction Knowledge Graphs (KGs), which store facts as triples in the form of (subject entity, relation, object entity), benefit various NLP applications (Lan and Jiang, 2020; Wang et al., 2019b; He et al., 2017). However, existing KGs face with serious incom- pleteness despite of their large scales. Therefore, KG completion, which aims to reason missing facts based on existing triples, has been an important re- search area. The past decade has witnessed the rise of embedding-based reasoning methods on KGs (Bor- des et al., 2013; Yang et al., 2014; Balaˇzevi´c et al., 2019). However, due to their black-box nature, these methods cannot provide interpretations for a specific prediction (Ji et al., 2020; Sadeghian et al., 2019). Recently, there has been growing interest in using multi-hop reasoning to improve the inter- pretability (Gardner et al., 2013; Rockt¨aschel and Riedel, 2017). This approach usually adopts Rein- forcement Learning (RL) to find a reasoning path (Xiong et al., 2017; Das et al., 2018; Hildebrandt et al., 2020). Starting from the query entity, the RL-based model sequentially selects an outgoing edge and transits to a new entity until it arrives at the target. However, due to the complexity of the KG, the number of paths grows exponentially when the rea- soning hop increases. Most of paths cannot arrive at correct answers, and cannot receive a none-zero re- ward, which is also called the “sparse reward prob- lem” (Nair et al., 2018). Moreover, since golden paths are not available in the training process, the RL-based model may coincidentally reach the tar- get via a meaningless path (i.e. spurious paths). Take the query (Captain America, director, ?) as an instance. Although the path (Captain America, country, US, lives in−1, Peter Farrelly), can arrive at …

Content truncated to 500 words