SimKGC: Simple Contrastive Knowledge Graph Completion with Pre-trained Language Models

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 4281 - 4294 May 22-27, 2022 c⃝2022 Association for Computational Linguistics SimKGC: Simple Contrastive Knowledge Graph Completion with Pre-trained Language Models Liang Wang1∗and Wei Zhao2 and Zhuoyu Wei2 and Jingming Liu2 1Microsoft Research Asia 2Yuanfudao AI Lab, Beijing, China wangliang@microsoft.com {zhaowei01,weizhuoyu,liujm}@yuanfudao.com Abstract Knowledge graph completion (KGC) aims to reason over known facts and infer the miss- ing links. Text-based methods such as KG- BERT (Yao et al., 2019) learn entity represen- tations from natural language descriptions, and have the potential for inductive KGC. How- ever, the performance of text-based methods still largely lag behind graph embedding-based methods like TransE (Bordes et al., 2013) and RotatE (Sun et al., 2019b). In this pa- per, we identify that the key issue is efficient contrastive learning. To improve the learning efficiency, we introduce three types of nega- tives: in-batch negatives, pre-batch negatives, and self-negatives which act as a simple form of hard negatives. Combined with InfoNCE loss, our proposed model SimKGC can sub- stantially outperform embedding-based meth- ods on several benchmark datasets. In terms of mean reciprocal rank (MRR), we advance the state-of-the-art by +19% on WN18RR, +6.8% on the Wikidata5M transductive set- ting, and +22% on the Wikidata5M inductive setting. Thorough analyses are conducted to gain insights into each component. Our code is available at https://github.com/ intfloat/SimKGC. 1 Introduction Large-scale knowledge graphs (KGs) are important components for knowledge-intensive applications, such as question answering (Sun et al., 2019a), recommender systems (Huang et al., 2018), and in- telligent conversational agents (Dinan et al., 2019) etc. KGs usually consist of a set of triples (h, r, t), where h is the head entity, r is the relation, and t is the tail entity. Popular public KGs include Freebase (Bollacker et al., 2008), Wikidata (Vran- deˇci´c and Krötzsch, 2014), YAGO (Suchanek et al., 2007), ConceptNet (Speer et al., 2017), and Word- Net (Miller, 1992) etc. Despite their usefulness ∗Work done while at Yuanfudao AI Lab. in practice, they are often incomplete. Knowledge graph completion (KGC) techniques are necessary for the automatic construction and verification of knowledge graphs. country of citizenship continent named after instance of Mount Everest is Earth's highest mountain above sea level... Mount Everest Asia is Earth's largest and most populous continent... Asia A mountain is an elevated portion of the Earth's crust… Mountain ... was a British surveyor and geographer who... George Everest The United Kingdom ... is a sovereign country in north- western Europe... United Kingdom Figure 1: An example of knowledge graph. Each entity has its name and textual descriptions. Existing KGC methods can be categorized into two families: embedding-based and text-based methods. Embedding-based methods map each entity and relation into a low-dimensional vector, without using any side information such as entity descriptions. This family includes TransE (Bor- des et al., 2013), TransH (Wang et al., 2014), Ro- tatE (Sun et al., 2019b), and TuckER (Balaze- vic et al., 2019) etc. By comparison, text-based …

Content truncated to 500 words