ZELDA: A Comprehensive Benchmark for Supervised Entity Disambiguation

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2061–2072 May 2-6, 2023 ©2023 Association for Computational Linguistics ZELDA: A Comprehensive Benchmark for Supervised Entity Disambiguation Marcel Milich Humboldt-Universitat zu Berlin marcelmilch@gmx.de Alan Akbik Humboldt-Universitat zu Berlin alan.akbik@hu-berlin.de Abstract Entity disambiguation (ED) is the task of disambiguating named entity mentions in text to unique entries in a knowledge base. Due to its industrial relevance, as well as current progress in leveraging pre-trained language models, a multitude of ED approaches have been proposed in recent years. However, we observe a severe lack of uniformity across experimental setups in current ED work, rendering a direct comparison of approaches based solely on reported numbers impossible: Current approaches widely differ in the data set used to train, the size of the covered entity vocabulary, and the usage of additional signals such as candidate lists. To address this issue, we present ZELDA, a novel entity disambigua- tion benchmark that includes a unified training data set, entity vocabulary, candidate lists, as well as challenging evaluation splits covering 8 different domains. We illustrate its design and construction, and present experiments in which we train and compare current state-of-the-art approaches on our benchmark. To encourage greater direct comparability in the entity disambiguation domain, we open source our benchmark at https: //github.com/flairNLP/zelda. 1 Introduction Entity disambiguation (ED) is the task of disam- biguating textual mentions of entities to a corre- sponding unique entry in a knowledge base. For instance, the entity mention "NBA" might refer to one of several organizations with this abbrevia- tion, such as "National Basketball Association" or "National Boxing Association". ED resolves these ambiguities and creates links between a knowledge base of unique entities and the various ways an entity may be referred to in text. It is the core component in the larger task of entity linking (EL), which includes the identification of entity mentions in text, often handled by a named entity recognition (NER) system. Recent progress in the field is driven by advances in large language models (Shen et al., 2021; Sevgili et al., 2022), pushing the scores on standard evalua- tion datasets to new heights. These models are typ- ically trained in a supervised manner. Unlike many other NLP tasks with relatively few target classes, such as sentiment analysis or part-of-speech tag- ging, ED may have millions of target classes, since each entity in a knowledge base is modeled as a distinct class. Accordingly, most current state-of- the-art ED approaches are trained over very large amounts of annotated text data that often is auto- matically derived from Wikipedia. Lack of uniformity in experimental setup. How- ever, while a number of standard evaluation datasets exist to measure final ED accuracy, such as the AIDA-B test split of the popular AIDA dataset for newswire data (Hoffart et al., 2011), we observe that no such standardization exists for the data used to train ED systems. To illustrate this disparity, refer to Table 1 for an overview of current …

Content truncated to 500 words