KnowledgeNet: A Benchmark Dataset for Knowledge Base Population

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 749–758, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 749 KnowledgeNet: A Benchmark Dataset for Knowledge Base Population Filipe Mesquita, Matteo Cannaviccio, Jordan Schmidek, Paramita Mirza Diffbot Technologies Corp. Menlo Park, California {filipe,matteo,jay,paramita}@diffbot.com Denilson Barbosa Department of Computing Science University of Alberta Edmonton, Canada denilson@ualberta.ca Abstract KnowledgeNet is a benchmark dataset for the task of automatically populating a knowledge base (Wikidata) with facts expressed in natural language text on the web. KnowledgeNet pro- vides text exhaustively annotated with facts, thus enabling the holistic end-to-end evalua- tion of knowledge base population systems as a whole, unlike previous benchmarks that are more suitable for the evaluation of individ- ual subcomponents (e.g., entity linking, rela- tion extraction). We discuss five baseline ap- proaches, where the best approach achieves an F1 score of 0.50, significantly outperforming a traditional approach by 79% (0.28). However, our best baseline is far from reaching human performance (0.82), indicating our dataset is challenging. The KnowledgeNet dataset and baselines are available at https://github. com/diffbot/knowledge-net 1 Introduction Knowledge Bases (KBs) are valuable resources for developing intelligent applications, including search, question answering, data integration, and recommendation systems. High-quality KBs still rely almost exclusively on human-curated struc- tured or semi-structured data. Such a reliance on human curation is a major obstacle to the creation of comprehensive, always-up-to-date KBs. KB population (KBP) is the task of automati- cally augmenting a KB with new facts. Tradition- ally, KBP has been tackled with datasets for in- dividual components to be arranged in a pipeline, typically: (1) entity discovery and linking (Ji et al., 2017; Shen et al., 2015) and (2) relation extrac- tion (Angeli et al., 2015; Zhang et al., 2017). En- tity discovery and linking seeks to recognize and disambiguate proper names in text that refer to en- tities (e.g., people, organizations and locations) by linking them to a reference KB. Relation extrac- tion seeks to detect facts involving two entities (or an entity and a literal, such as a number or date). KnowledgeNet is a benchmark dataset for pop- ulating a KB (Wikidata) with facts expressed in natural language on the web. KnowledgeNet facts are of the form (subject; property; object), where subject and object are linked to Wikidata. For in- stance, the dataset contains text expressing the fact (Gennaro Basile1; RESIDENCE; Moravia2), in the passage: “Gennaro Basile was an Italian painter, born in Naples but active in the German- speaking countries. He settled at Br¨unn, in Moravia, and lived about 1756...” KnowledgeNet’s main goal is to evaluate the overall task of KBP rather than evaluating its sub- components in separate. We refer to this type of evaluation as end-to-end. The dataset supports the end-to-end evaluation of KBP systems by exhaus- tively annotating all facts in a sentence. For in- stance, the dataset contains all RESIDENCE facts (two) from the sentence “He settled at Br¨unn, in Moravia, and …

Content truncated to 500 words