Do Pre-trained Models Benefit Knowledge Graph Completion? A Reliable Evaluation and a Reasonable Approach

Findings of the Association for Computational Linguistics: ACL 2022, pages 3570 - 3581 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Do Pre-trained Models Benefit Knowledge Graph Completion? A Reliable Evaluation and a Reasonable Approach Xin Lv1,2, Yankai Lin3, Yixin Cao4, Lei Hou1,2∗, Juanzi Li1,2 Zhiyuan Liu1,2, Peng Li5† , Jie Zhou3 1Department of Computer Science and Technology, BNRist 2KIRC, Institute for Artificial Intelligence, Tsinghua University, Beijing 100084, China 3Pattern Recognition Center, WeChat AI, Tencent Inc., China 4Singapore Management University, Singapore 5Institute for AI Industry Research (AIR), Tsinghua University, China lv-x18@mails.tsinghua.edu.cn, houlei@tsinghua.edu.cn Abstract In recent years, pre-trained language models (PLMs) have been shown to capture factual knowledge from massive texts, which encour- ages the proposal of PLM-based knowledge graph completion (KGC) models. However, these models are still quite behind the SOTA KGC models in terms of performance. In this work, we find two main reasons for the weak performance: (1) Inaccurate evaluation setting. The evaluation setting under the closed- world assumption (CWA) may underestimate the PLM-based KGC models since they intro- duce more external knowledge; (2) Inappro- priate utilization of PLMs. Most PLM-based KGC models simply splice the labels of entities and relations as inputs, leading to incoherent sentences that do not take full advantage of the implicit knowledge in PLMs. To alleviate these problems, we highlight a more accurate evalua- tion setting under the open-world assumption (OWA), which manually checks the correctness of knowledge that is not in KGs. Moreover, mo- tivated by prompt tuning, we propose a novel PLM-based KGC model named PKGC. The basic idea is to convert each triple and its sup- port information into natural prompt sentences, which are further fed into PLMs for classifica- tion. Experiment results on two KGC datasets demonstrate OWA is more reliable for evaluat- ing KGC, especially on the link prediction, and the effectiveness of our PKCG model on both CWA and OWA settings. 1 Introduction Knowledge graph (KG) has gradually become the cornerstone of many Natural Language Processing (NLP) tasks (Cui et al., 2017; Zhou et al., 2018), as one of the most effective ways to represent world ∗ Corresponding Author † Part of the work was done while Peng Li was working at Tencent. Triple Query: (England, contains, ?) PLM-based KGC Model 1. Lancashire 2. Suffolk 3. Sunderland ... 14. Pontefract 15. Dundalk Closed-world Assumption Open-world Assumption Figure 1: Evaluation results for link prediction under different settings. The bolded entities in the dashed box are all correct answers, but only the red entities are considered correct under the closed-world assumption. knowledge. To improve the coverage, researchers have automated knowledge extraction techniques or relied on collaborative editing, while these KGs still hardly cover the massive emerging knowledge in the real world. This problem motivates knowl- edge graph completion (KGC), the task of predict- ing missing links through understanding existing structures in KGs. Soon sweeping across the entire NLP field, the potential of pre-trained language models (PLMs) for KGC has attracted much attention. Petroni et al. (2019); Shin et al. …

Content truncated to 500 words