KGLM: Integrating Knowledge Graph Structure in Language Models for Link Prediction

Proceedings of the The 12th Joint Conference on Lexical and Computational Semantics (*SEM 2023), pages 217–224 July 13-14, 2023 ©2023 Association for Computational Linguistics KGLM: Integrating Knowledge Graph Structure in Language Models for Link Prediction Jason Youn1,2,3 and Ilias Tagkopoulos 1,2,3 1 Department of Computer Science, University of California, Davis, CA 95616, USA. 2 Genome Center, University of California, Davis, CA 95616, USA. 3 USDA/NSF AI Institute for Next Generation Food Systems (AIFS), University of California, Davis, CA 95616, USA. {jyoun, itagkopoulos}@ucdavis.edu Abstract The ability of knowledge graphs to represent complex relationships at scale has led to their adoption for various needs including knowl- edge representation, question-answering, and recommendation systems. Knowledge graphs are often incomplete in the information they represent, necessitating the need for knowledge graph completion tasks. Pre-trained and fine- tuned language models have shown promise in these tasks although these models ignore the intrinsic information encoded in the knowledge graph, namely the entity and relation types. In this work, we propose the Knowledge Graph Language Model (KGLM) architecture, where we introduce a new entity/relation embedding layer that learns to differentiate distinctive en- tity and relation types, therefore allowing the model to learn the structure of the knowledge graph. In this work, we show that further pre- training the language models with this addi- tional embedding layer using the triples ex- tracted from the knowledge graph, followed by the standard fine-tuning phase sets a new state-of-the-art performance for the link predic- tion task on the benchmark datasets. 1 Introduction Knowledge graph (KG) is defined as a directed, multi-relational graph where entities (nodes) are connected with one or more relations (edges) (Wang et al., 2017). It is represented with a set of triples, where a triple consists of (head entity, relation, tail entity) or (h, r, t) for short, for ex- ample (Bill Gates, founderOf, Microsoft) as shown in Figure 1. Due to their effectiveness in iden- tifying patterns among data and gaining insights into the mechanisms of action, associations, and testable hypotheses (Li and Chen, 2014; Silvescu et al., 2012), both manually curated KGs like DB- pedia (Auer et al., 2007), WordNet (Miller, 1998), KIDS (Youn et al., 2022), and CARD (Alcock et al., 2020), and automatically curated ones like Free- Base (Bollacker et al., 2008), Knowledge Vault Bill Gates Microsoft Washington Melinda French Jennifer Gates founderOf locatedIn bornIn daughterOf daughterOf divorcedWith Figure 1: Sample knowledge graph with 6 triples. The graph contains three unique entity types (circle for per- son, triangle for company, and square for location) and 5 unique relation types or 10 if considering both the for- ward and inverse relations. The task of the knowledge graph completion is to complete the missing links in the graph, e.g., (Bill Gates, bornIn?, Washington) using the existing knowledge graph. (Dong et al., 2014), and NELL (Carlson et al., 2010) exist. However, these KGs often suffer from incompleteness. For example, 71% of the people in FreeBase have no known place of birth (West et al., 2014). To address this …

Content truncated to 500 words