Can We Predict New Facts with Open Knowledge Graph Embeddings? A Benchmark for Open Link Prediction

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2296–2308 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 2296 Can We Predict New Facts with Open Knowledge Graph Embeddings? A Benchmark for Open Link Prediction Samuel Broscheit, Kiril Gashteovski, Yanjie Wang, Rainer Gemulla Data and Web Science Group University of Mannheim, Germany broscheit@informatik.uni-mannheim.de, {k.gashteovski,ywang,rgemulla}@uni-mannheim.de, Abstract Open Information Extraction systems extract (“subject text”, “relation text”, “object text”) triples from raw text. Some triples are textual versions of facts, i.e., non-canonicalized men- tions of entities and relations. In this paper, we investigate whether it is possible to infer new facts directly from the open knowledge graph without any canonicalization or any supervi- sion from curated knowledge. For this pur- pose, we propose the open link prediction task, i.e., predicting test facts by completing (“sub- ject text”, “relation text”, ?) questions. An evaluation in such a setup raises the question if a correct prediction is actually a new fact that was induced by reasoning over the open knowl- edge graph or if it can be trivially explained. For example, facts can appear in different para- phrased textual variants, which can lead to test leakage. To this end, we propose an evaluation protocol and a methodology for creating the open link prediction benchmark OLPBENCH. We performed experiments with a prototypical knowledge graph embedding model for open link prediction. While the task is very chal- lenging, our results suggests that it is possible to predict genuinely new facts, which can not be trivially explained. 1 Introduction A knowledge graph (KG) (Hayes-Roth, 1983) is a set of (subject, relation, object)-triples, where the subject and object correspond to vertices, and rela- tions to labeled edges. In curated KGs, each triple is fully disambiguated against a fixed vocabulary of entities1 and relations. An application for KGs, for example, is the prob- lem of drug discovery based on bio-medical knowl- edge (Mohamed et al., 2019). The construction of a curated bio-medical KG, which is required for 1For brevity, “entities” denotes both entities (e.g. Prince) and concepts (e.g. musician) throughout the paper. “NBC Television” “NBC” “NBC-TV” NBC NewYorkCity Knowledge Graph Open Knowledge Graph “NYC” “New York City” Figure 1: Entities and relations in curated knowledge graphs vs. open knowledge graphs. such an approach, is challenging and constrained by the available amount of human effort and domain expertise. Many tools that could assist humans in KG construction (e.g., an entity linker) need a KG to begin with. Moreover, current methods for KG construction often rely on the rich structure of Wikipedia, such as links and infoboxes, which are not available for every domain. Therefore, we ask if it is possible to make predictions about, for exam- ple, new drug applications from raw text without the intermediate step of KG construction. Open information extraction systems (OIE) (Et- zioni et al., 2011) automatically extract (“sub- ject text”, “relation text”, “object text”)-triples from unstructured data such as text. We can view OIE data as an open knowledge graph …

Content truncated to 500 words