Knowledge Graph and Text Jointly Embedding

Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1591–1601, October 25-29, 2014, Doha, Qatar. c⃝2014 Association for Computational Linguistics Knowledge Graph and Text Jointly Embedding Zhen Wang†§ , Jianwen Zhang† , Jianlin Feng§ , Zheng Chen† †{v-zw,jiazhan,zhengc}@microsoft.com §{wangzh56@mail2,fengjlin@mail}.sysu.edu.cn †Microsoft Research §Sun Yat-sen University Abstract We examine the embedding approach to reason new relational facts from a large- scale knowledge graph and a text corpus. We propose a novel method of jointly em- bedding entities and words into the same continuous vector space. The embedding process attempts to preserve the relations between entities in the knowledge graph and the concurrences of words in the text corpus. Entity names and Wikipedia an- chors are utilized to align the embeddings of entities and words in the same space. Large scale experiments on Freebase and a Wikipedia/NY Times corpus show that jointly embedding brings promising improvement in the accuracy of predicting facts, compared to separately embedding knowledge graphs and text. Particularly, jointly embedding enables the prediction of facts containing entities out of the knowledge graph, which cannot be han- dled by previous embedding methods. At the same time, concerning the quality of the word embeddings, experiments on the analogical reasoning task show that jointly embedding is comparable to or slightly better than word2vec (Skip-Gram). 1 Introduction Knowledge graphs such as Freebase (Bollacker et al., 2008) and WordNet (Miller, 1995) have be- come important resources for many AI & NLP ap- plications such as Q & A. Generally, a knowledge graph is a collection of relational facts that are of- ten represented in the form of a triplet (head en- tity, relation, tail entity), e.g., “(Obama, Born-in, Honolulu)”. An urgent issue for knowledge graph- s is the coverage, e.g., even the largest knowledge graph of Freebase is still far from complete. Recently, targeting knowledge graph comple- tion, a promising paradigm of embedding was pro- posed, which is able to reason new facts only from the knowledge graph (Bordes et al., 2011; Bor- des et al., 2013; Socher et al., 2013; Wang et al., 2014). Generally, in this series of methods, each entity is represented as a k-dimensional vector and each relation is characterized by an operation in ℜk so that a candidate fact can be asserted by sim- ple vector operations. The embeddings are usually learnt by minimizing a global loss function of all the entities and relations in the knowledge graph. Thus, the vector of an entity may encode global information from the entire graph, and hence scor- ing a candidate fact by designed vector operations plays a similar role to long range “reasoning” in the graph. However, since this requires the vectors of both entities to score a candidate fact, this type of methods can only complete missing facts for which both entities exist in the knowledge graph. However, a missing fact often contains entities out of the knowledge graph (called out-of-kb for short in this paper), e.g., one or both entities are phras- es appearing in …

Content truncated to 500 words