A Framework for Adapting Pre-Trained Language Models to Knowledge Graph Completion

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5937 - 5955 December 7-11, 2022 ©2022 Association for Computational Linguistics A Framework for Adapting Pre-Trained Language Models to Knowledge Graph Completion Justin Lovelace∗ Computer Science Department Cornell University jl3353@cornell.edu Carolyn Penstein Rosé Language Technologies Institute Carnegie Mellon University cp3a@andrew.cmu.edu Abstract Recent work has demonstrated that entity rep- resentations can be extracted from pre-trained language models to develop knowledge graph completion models that are more robust to the naturally occurring sparsity found in knowl- edge graphs. In this work, we conduct a com- prehensive exploration of how to best extract and incorporate those embeddings into knowl- edge graph completion models. We explore the suitability of the extracted embeddings for direct use in entity ranking and introduce both unsupervised and supervised processing meth- ods that can lead to improved downstream per- formance. We then introduce supervised em- bedding extraction methods that can extract more informative representations. We then syn- thesize our findings and develop a knowledge graph completion model that significantly out- performs recent neural models. 1 1 Introduction Knowledge graphs (KG) are structured represen- tations of knowledge that contain a collection of factual relations between entities. KGs are valu- able resources with applications in different areas such as representation learning (Liu et al., 2018), question answering (Sun et al., 2019; Shen et al., 2019; Thirukovalluru et al., 2021), and entity link- ing (Thai et al., 2021). However, the difficulty of curating knowledge at scale means that existing KGs are highly in- complete. This has led to the widespread study of knowledge graph completion (KGC) which aims to develop automated solutions that can suggest new facts to add to the KG (Yang et al., 2015; Trouil- lon et al., 2016; Dettmers et al., 2018). KGC is typically formulated as ranking problem where an incomplete fact is used as a query to retrieve enti- ties that complete the fact. ∗ Work conducted while at Carnegie Mellon University. 1https://github.com/justinlovelace/ LM-KG-Completion Recent work has utilized pre-trained language models to develop approaches that are more robust to the naturally occurring sparsity within knowl- edge graphs. These approaches utilize textual en- tity descriptions to develop entity representations that are less reliant on graph connectivity. Such work either fine-tunes the language model directly during training to encode the entities (e.g. Yao et al. (2019)) or extracts a set of entity embed- dings prior to training which can then be used to train a KGC model using standard training proce- dures (e.g. Lovelace et al. (2021)). While fine-tuning language models often im- proves downstream performance (Rogers et al., 2020), it increases the computational overhead of computing entity representations. As a result, stan- dard KGC training procedures that involve evaluat- ing a large number of negative candidates for each positive instance are typically infeasible. Sampling only a small set of negative candidates enables training, but can harm performance. Approaches that extract entity embeddings prior to training (Lovelace et al., 2021; Wang et al., 2021a) do not …

Content truncated to 500 words