Sparsity and Noise: Where Knowledge Graph Embeddings Fall Short

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1751–1756 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Sparsity and Noise: Where Knowledge Graph Embeddings Fall Short Jay Pujara and Eriq Augustine and Lise Getoor Department of Computer Science University of California, Santa Cruz jay@cs.umd.edu, eaugustine@ucsc.edu, getoor@soe.ucsc.edu Abstract Knowledge graph (KG) embedding techniques use structured relation- ships between entities to learn low- dimensional representations of entities and relations. One prominent goal of these approaches is to improve the quality of knowledge graphs by remov- ing errors and adding missing facts. Surprisingly, most embedding tech- niques have been evaluated on bench- mark datasets consisting of dense and reliable subsets of human-curated KGs, which tend to be fairly complete and have few errors. In this paper, we consider the problem of applying em- bedding techniques to KGs extracted from text, which are often incomplete and contain errors. We compare the sparsity and unreliability of different KGs and perform empirical experi- ments demonstrating how embedding approaches degrade as sparsity and un- reliability increase. 1 Introduction Recently knowledge graphs (KGs), structured representations of knowledge bases, have be- come an essential component of systems that perform question-answering (Berant et al., 2013), provide decision support, and enable exploration and discovery (Dong et al., 2014). Initial efforts to create KGs focused on struc- tured information sources or relied extensively on manual curation. However, the diversity of knowledge available on resources like the World Wide Web have spurred many projects that tackle the more difficult task of automat- ically constructing KGs (Nickel et al., 2016a). Unfortunately, information extraction ap- proaches for KG construction must overcome complex, unreliable, and incomplete data. Many machine learning methods have been proposed to address the challenge of cleaning and completing KGs. One popular class of methods learn embeddings that translate en- tities and relationships into a latent subspace, then use this latent representation to derive additional, unobserved facts and score exist- ing facts (Bordes et al., 2013; Wang et al., 2014; Lin et al., 2015). Embedding methods have shown state-of- the-art results on several benchmark datasets. However, by construction, these benchmark datasets differ from data in real KGs. First, benchmark datasets have largely been re- stricted to the most frequently occurring en- tities in the KG. However in most KGs, en- tities are associated with a sparse set of ob- servations. Second, benchmark datasets con- sist only of highly reliable facts from cu- rated knowledge bases. In contrast, many KG construction projects extract knowledge from noisy data such as text or images, which in- troduces unreliable information. In this paper, we evaluate popular KG em- bedding approaches on KGs that have sparse entities and unreliable candidate facts. We ap- ply embedding methods to an extracted KG and modify existing benchmarks by varying the sparsity and reliability of training data used to learn embedding models. Using this suite of datasets, we characterize where em- bedding approaches are successful and the con- ditions that result in degrading results. Based …

Content truncated to 500 words