Evaluating Entity Linking: An Analysis of Current Benchmark Datasets and a Roadmap for Doing a Better Job

Evaluating Entity Linking: An Analysis of Current Benchmark Datasets and a Roadmap for Doing a Better Job Marieke van Erp♦, Pablo N. Mendes∗, Heiko Paulheim∓, Filip Ilievski♦, Julien Plu◦, Giuseppe Rizzo‡, J¨org Waitelonis∞ ♦Vrije Universiteit Amsterdam, Netherlands {marieke.van.erp,f.ilievski}@vu.nl ∗IBM Research Almaden, USA pnmendes@us.ibm.com ∓Data and Web Science Group, University of Mannheim, Germany heiko@informatik.uni-mannheim.de ◦EURECOM, France plu@eurecom.fr ‡ISMB, Italy giuseppe.rizzo@ismb.it ∞Hasso-Plattner-Institute Potsdam, Germany joerg.waitelonis@hpi.de Abstract Entity linking has become a popular task in both natural language processing and semantic web communities. However, we find that the benchmark datasets for entity linking tasks do not accurately evaluate entity linking systems. In this paper, we aim to chart the strengths and weaknesses of current benchmark datasets and sketch a roadmap for the community to devise better benchmark datasets. Keywords: entity linking, evaluation, benchmark 1. Introduction Since the 90’s, recognizing and linking entities has been a popular research topic. Initially, research attempts focused on identifying and classifying atomic information units in text (Named Entity Recognition and Classification). Later on, research into linking mentions to external resources of knowledge bases referents (Named Entity Linking) was introduced. This triggered numerous research initiatives such as CoNLL (Tjong Kim Sang and De Meulder, 2003), ACE (Doddington et al., 2004), TAC-KBP (McNamee, 2009), NEEL (Cano et al., 2014; Rizzo et al., 2015), Se- mEval (Moro and Navigli, 2015), or ERD (Carmel et al., 2014), aiming at building common and general benchmark datasets to test, adapt, and improve entity recognition and linking. Benchmark datasets have often been treated as black boxes by published research, making it difficult to interpret effi- cacy improvements in terms of individual contributions of algorithms and/or labeled data. This scattered landscape of datasets and measures leads to a misleading interpretability of the experimental results, which makes the performance evaluation of novel approaches against the state of the art rather difficult and open to several interpretations and ques- tions. In this paper, we aim to highlight the strengths and weak- nesses of common benchmark datasets developed for eval- uation of entity linking systems. We inspect heterogeneous benchmark datasets in terms of genre such as newswire, blog posts, and microblog posts. Dataset characteristics have been previously reviewed in (Ling et al., 2015) as part of the overall analysis of the entity linking task, in (Stein- metz et al., 2013) with focus on candidate generation, and in GERBIL (Usbeck et al., 2015). Our work is comple- mentary to all, as we focus on deepening the analysis of the benchmark datasets. We approach this by highlighting strengths and weaknesses of these datasets through quan- tifiable aspects such as entity overlap, dominance, and pop- ularity. We hope that this work may foster metric-aware and less biased benchmark datasets to be created, as well as it may stimulate a more sensitive discussion of results produced using those benchmarks. The paper is organized as follows: Section 2. gives an overview of the benchmark datasets we investigate. We de- tail their characteristics in Section 3. In Section 4. we report insights and key interpretations, …

Content truncated to 500 words