KORE 50ˆDYWC: An Evaluation Data Set for Entity Linking Based on DBpedia, YAGO, Wikidata, and Crunchbase

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 2389–2395 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 2389 KORE 50DYWC: An Evaluation Data Set for Entity Linking Based on DBpedia, YAGO, Wikidata, and Crunchbase Kristian Noullet, Rico Mix, Michael F¨arber Karlsruhe Institute of Technology (KIT) kristian.noullet@kit.edu, ucvde@student.kit.edu, michael.faerber@kit.edu Abstract A major domain of research in natural language processing is named entity recognition and disambiguation (NERD). One of the main ways of attempting to achieve this goal is through use of Semantic Web technologies and its structured data formats. Due to the nature of structured data, information can be extracted more easily, therewith allowing for the creation of knowledge graphs. In order to properly evaluate a NERD system, gold standard data sets are required. A plethora of different evaluation data sets exists, mostly relying on either Wikipedia or DBpedia. Therefore, we have extended a widely-used gold standard data set, KORE 50, to not only accommodate NERD tasks for DBpedia, but also for YAGO, Wikidata and Crunchbase. As such, our data set, KORE 50DYWC, allows for a broader spectrum of evaluation. Among others, the knowledge graph agnosticity of NERD systems may be evaluated which, to the best of our knowledge, was not possible until now for this number of knowledge graphs. Keywords: Data Sets, Entity Linking, Knowledge Graph, Text Annotation, NLP Interchange Format 1. Introduction Automatic extraction of semantic information from texts is an open problem in natural language processing (NLP). It requires appropriate techniques such as named entity recog- nition and disambiguation (NERD), which is the process of extracting named entities from texts and linking them to a knowledge graph (KG). In the past, several methods have been developed for this task, such as DBpedia Spot- light (Mendes et al., 2011), Babelfy (Moro et al., 2014a; Moro et al., 2014b), MAG (Moussallem et al., 2017), X-LiSA (Zhang and Rettinger, 2014), TagMe (Ferragina and Scaiella, 2010) and AGDISTIS (Usbeck et al., 2014). These methods face a lot of challenges, such as ambigu- ity among short pieces of text – corresponding to a poten- tially large number of entities (known as surface forms) – and overall hard-to-disambiguate mentions like forenames or typographical errors in texts. Thus, it is important to provide a holistic evaluation of these methods in order to identify potential strengths and weaknesses which help to improve them. In the past, there have been a number of approaches aim- ing for “proper” entity linking system evaluation, such as CoNLL (Sang and Meulder, 2003), ACE (Doddington et al., 2004) or NEEL (Rizzo et al., 2015). Besides the devel- opment of entity linking systems, such approaches focus on the creation of general benchmark data sets which test and improve annotation methods and therefore entity linking. For evaluating NERD, typically gold standards are used which require manual annotation making their creation a time-consuming and complex task. However, current state- of-the-art evaluation data sets are subject to limitations: 1. The majority of evaluation data sets …

Content truncated to 500 words