Farewell Freebase: Migrating the SimpleQuestions Dataset to DBpedia

Proceedings of the 27th International Conference on Computational Linguistics, pages 2093–2103 Santa Fe, New Mexico, USA, August 20-26, 2018. 2093 Farewell Freebase: Migrating the SimpleQuestions Dataset to DBpedia Michael Azmy∗, Peng Shi∗, Jimmy Lin, and Ihab F. Ilyas David R. Cheriton School of Computer Science University of Waterloo Waterloo, Ontario, Canada {mwazmy,p8shi,jimmylin,ilyas}@uwaterloo.ca Abstract Question answering over knowledge graphs is an important problem of interest both commer- cially and academically. There is substantial interest in the class of natural language questions that can be answered via the lookup of a single fact, driven by the availability of the popular SIMPLEQUESTIONS dataset. The problem with this dataset, however, is that answer triples are provided from Freebase, which has been defunct for several years. As a result, it is difficult to build “real-world” question answering systems that are operationally deployable. Furthermore, a defunct knowledge graph means that much of the infrastructure for querying, browsing, and ma- nipulating triples no longer exists. To address this problem, we present SIMPLEDBPEDIAQA, a new benchmark dataset for simple question answering over knowledge graphs that was created by mapping SIMPLEQUESTIONS entities and predicates from Freebase to DBpedia. Although this mapping is conceptually straightforward, there are a number of nuances that make the task non-trivial, owing to the different conceptual organizations of the two knowledge graphs. To lay the foundation for future research using this dataset, we leverage recent work to provide simple yet strong baselines with and without neural networks. 1 Introduction Question answering over knowledge graphs is an important problem at the intersection of multiple re- search communities, with many commercial deployments. To ensure continued progress, it is important that open and relevant benchmarks are available to support the comparison of various techniques. In this paper, we focus on the class of questions that can be answered by a single triple (i.e., fact) from a knowledge graph. For example, the question “What type of music is on the album Phenomenon?” can be answered via the lookup of a simple fact—in this case, the “genre” property of the entity “Phenomenon”. Analysis of an existing benchmark dataset (Yao, 2015) and real-world user questions (Dai et al., 2016; Ture and Jojic, 2017) show that such questions cover a broad range of users’ needs. The SIMPLEQUESTIONS dataset (Bordes et al., 2015) has emerged as the de facto benchmark for evaluating these simple questions over knowledge graphs. However, there is one major deficiency with this resource: the answers draw from Freebase. Unfortunately, Freebase is defunct and no longer main- tained. This creates a number of insurmountable challenges: First, because the knowledge graph is stale, it is no longer possible to build a “real-world” operational QA system using models trained on SIMPLEQUESTIONS. Second, a defunct knowledge graph means that researchers must develop custom infrastructure for querying, browsing, and manipulating the graph. Thus, we are not able to leverage multiple cooperative and interchangeable service APIs that are deployed and maintained by different parties—which is the strength of the broader “open linked data” ecosystem. While …

Content truncated to 500 words