Fine-Grained Evaluation for Entity Linking

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 718–727, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 718 Fine-Grained Evaluation for Entity Linking Henry Rosales-M´endez, Aidan Hogan, Barbara Poblete IMFD Chile & Department of Computer Science, University of Chile {hrosales,ahogan,bpoblete}@dcc.uchile.cl Abstract The Entity Linking (EL) task identifies entity mentions in a text corpus and associates them with an unambiguous identifier in a Knowl- edge Base. While much work has been done on the topic, we first present the results of a survey that reveal a lack of consensus in the community regarding what forms of men- tions in a text and what forms of links the EL task should consider. We argue that no one definition of the Entity Linking task fits all, and rather propose a fine-grained cate- gorization of different types of entity men- tions and links. We then re-annotate three EL benchmark datasets – ACE2004, KORE50, and VoxEL – with respect to these categories. We propose a fuzzy recall metric to address the lack of consensus and conclude with fine- grained evaluation results comparing a selec- tion of online EL systems. 1 Introduction Entity Linking (EL) is an Information Extraction task whose goal is to identify mentions of entities in a text and to link each mention to an unambigu- ous identifier in a Knowledge Base (KB) such as Wikipedia, BabelNet (Moro et al., 2014), DBpe- dia (Lehmann et al., 2015), Freebase (Bollacker et al., 2008), Wikidata (Vrandeˇci´c and Kr¨otzsch, 2014) or YAGO (Rebele et al., 2016) (among others). The results of this task offer a practi- cal bridge between unstructured text and struc- tured KBs, where EL has applications for semantic search, document classification, relation extrac- tion, and more besides (Wu et al., 2018). While a broad number of EL techniques and systems have been proposed in recent years (Wu et al., 2018), a number of authors have noted that there is a lack of consensus on the fundamental question of what kinds of mentions in a text an EL system should link to which identifiers in the KB (Ling et al., 2015; Waitelonis et al., 2016; Jha et al., 2017; Rosales-M´endez et al., 2018b). A closely related task to EL is that of Named Entity Recognition (NER), which identifies men- tions of (named) entities in a task, but without link- ing the mention with a KB identifier. The types of entities that the NER task should target were defined at the Message Understanding Confer- ence 6 (MUC-6) (Grishman and Sundheim, 1996), specifically entities corresponding to types such as Person, Organization, Place and other Numer- ical/Temporal expressions. While this provided a consensus for evaluating NER systems, some au- thors noted that such a categorization is coarse- grained and proposed finer-grained classification mechanisms (Fleischman and Hovy, 2002). In the context of EL, target Knowledge Bases will often contain entities from a wide variety of classes, including Movies, Products, Events, …

Content truncated to 500 words