KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation

KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation Xiaozhi Wang1, Tianyu Gao3, Zhaocheng Zhu4,5, Zhengyan Zhang1 Zhiyuan Liu1,2∗, Juanzi Li1,2, and Jian Tang4,6,7∗ 1Department of CST, BNRist; 2KIRC, Institute for AI, Tsinghua University, Beijing, China {wangxz20,zy-z19}@mails.tsinghua.edu.cn {liuzy,lijuanzi}@tsinghua.edu.cn 3Department of Computer Science, Princeton University, Princeton, NJ, USA tianyug@princeton.edu 4Mila - Qu´ebec AI Institute; 5Univesit´e de Montr´eal; 6HEC, Montr´eal, Canada zhaocheng.zhu@umontreal.ca, jian.tang@hec.ca 7CIFAR AI Research Chair Abstract Pre-trained language representation models (PLMs) cannot well capture factual knowledge from text. In contrast, knowledge embed- ding (KE) methods can effectively represent the relational facts in knowledge graphs (KGs) with informative entity embeddings, but conventional KE models cannot take full advantage of the abundant textual informa- tion. In this paper, we propose a unified model for Knowledge Embedding and Pre-trained LanguagE Representation (KEPLER), which can not only better integrate factual knowl- edge into PLMs but also produce effective text-enhanced KE with the strong PLMs. In KEPLER, we encode textual entity descrip- tions with a PLM as their embeddings, and then jointly optimize the KE and language modeling objectives. Experimental results show that KEPLER achieves state-of-the-art performances on various NLP tasks, and also works remarkably well as an inductive KE model on KG link prediction. Furthermore, for pre-training and evaluating KEPLER, we con- struct Wikidata5M1, a large-scale KG dataset with aligned entity descriptions, and bench- mark state-of-the-art KE methods on it. It shall serve as a new KE benchmark and facilitate the research on large KG, inductive KE, and KG with text. The source code can be obtained from https://github.com/THU-KEG /KEPLER. ∗Correspondence to: Z. Liu and J. Tang. 1https://deepgraphlearning.github.io /project/wikidata5m. 1 Introduction Recent pre-trained language representation mod- els (PLMs) such as BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019c) learn effective language representation from large-scale unstruc- tured corpora with language modeling objectives and have achieved superior performances on var- ious natural language processing (NLP) tasks. Existing PLMs learn useful linguistic knowl- edge from unlabeled text (Liu et al., 2019a), but they generally cannot capture the world facts well, which are typically sparse and have complex forms in text (Petroni et al., 2019; Logan et al., 2019). By contrast, knowledge graphs (KGs) contain extensive structural facts, and knowledge embed- ding (KE) methods (Bordes et al., 2013; Yang et al., 2015; Sun et al., 2019) can effectively embed them into continuous entity and relation embeddings. These embeddings can not only help with the KG completion but also benefit various NLP applications (Yang and Mitchell, 2017; Zaremoodi et al., 2018; Han et al., 2018a). As shown in Figure 1, textual entity descriptions contain abundant information. Intuitively, KE methods can provide factual knowledge for PLMs, while the informative text data can also benefit KE. Inspired by Xie et al. (2016), we use entity descriptions to bridge the gap between KE and PLM, and align the semantic space of text to the symbol space of KGs (Logeswaran et al., 2019). We propose KEPLER, a unified model for Knowledge Embedding and Pre-trained LanguagE Representation. We encode …

Content truncated to 500 words