Are Missing Links Predictable? An Inferential Benchmark for Knowledge Graph Completion

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 6855–6865 August 1–6, 2021. ©2021 Association for Computational Linguistics 6855 Are Missing Links Predictable? An Inferential Benchmark for Knowledge Graph Completion Yixin Cao12 Xiang Ji1 Xin Lv3 Juanzi Li3 Yonggang Wen1 Hanwang Zhang1 1School of CSE, Nanyang Technological University, Singapore 2S-Lab, Nanyang Technological University, Singapore 3Department of CST, Tsinghua University, Beijing, China caoyixin2011@gmail.com, lv-x18@mails.tsinghua.edu.cn lijuanzi@tsinghua.edu.cn, {ygwen,hanwangzhang}@ntu.edu.sg Abstract We present InferWiki, a Knowledge Graph Completion (KGC) dataset that improves upon existing benchmarks in inferential ability, as- sumptions, and patterns. First, each testing sample is predictable with supportive data in the training set. To ensure it, we propose to utilize rule-guided train/test generation, in- stead of conventional random split. Second, InferWiki initiates the evaluation following the open-world assumption and improves the in- ferential difficulty of the closed-world assump- tion, by providing manually annotated nega- tive and unknown triples. Third, we include various inference patterns (e.g., reasoning path length and types) for comprehensive evalua- tion. In experiments, we curate two settings of InferWiki varying in sizes and structures, and apply the construction process on CoDEx as comparative datasets. The results and em- pirical analyses demonstrate the necessity and high-quality of InferWiki. Nevertheless, the performance gap among various inferential as- sumptions and patterns presents the difficulty and inspires future research direction. Our datasets can be found in https://github. com/TaoMiner/inferwiki. 1 Introduction Knowledge Graph Completion (KGC) aims to pre- dict missing links in KG by inferring new knowl- edge from existing ones. Attributed to its reasoning ability, KGC models are crucial in alleviating the KG’s incompleteness issue and benefiting many downstream applications, such as recommenda- tion (Cao et al., 2019b) and information extrac- tion (Hu et al., 2021; Cao et al., 2020a). However, the KGC performance on existing benchmarks are still unsatisfactory — 0.51 Hit Ratio@1 and 187 Mean Rank of the top-ranked model (Wang et al., 2019) on the widely used FB15k237 (Toutanova and Chen, 2015). Do we have a slow progress of Head Predicate Tail Test David location ? (Ans: Florida) Train David placeOfBirth Atlanta David nationality U.S.A. Test Zurich travelMonth ? (Ans: October) Train Zurich travelMonth Jan., Feb., Mar., Apr., May., Jun., Jul., Aug., Sep., Nov., Dec. Table 1: Low-quality examples in FB15k237. We only present related triples. Ans denotes the missing entity. models (Akrami et al., 2020)? Or should we blame for the low-quality of benchmarks? In this paper, we re-think the task of KGC and construct a new benchmark dubbed InferWiki that highlights three fundamental objectives: Test triples should be inferential: this is the es- sential requirement of KGC. Each test triple should have supportive samples in the train set. How- ever, we observe two major issues of current KGC datasets: unpredictable and meaningless test triples, which may hinder evaluating and advancing state- of-the-arts. As shown in Table 1, the first example of inferring the location for David (i.e., Florida) is even impossible for humans — not …

Content truncated to 500 words