In Media Res: A Corpus for Evaluating Named Entity Linking with Creative Works

Proceedings of the 24th Conference on Computational Natural Language Learning, pages 355–364 Online, November 19-20, 2020. c⃝2020 Association for Computational Linguistics https://doi.org/10.18653/v1/P17 355 In Media Res: A Corpus for Evaluating Named Entity Linking with Creative Works Adrian M.P. Bra¸soveanu MODUL Technology GmbH Vienna, Austria adrian.brasoveanu@ modul.ac.at Albert Weichselbraun Fachhochschule Graubünden Chur, Switzerland albert.weichselbraun@ fhgr.ch Lyndon J.B. Nixon MODUL Technology GmbH Vienna, Austria lyndon.nixon@ modul.ac.at Abstract Annotation styles express guidelines that di- rect human annotators by explicitly stating the rules to follow when creating gold standard an- notations of text corpora. These guidelines not only shape the gold standards they help create, but also influence the training and evaluation of Named Entity Linking (NEL) tools, since different annotation styles correspond to diver- gent views on the entities present in a docu- ment. Such divergence is particularly relevant for texts from the media domain containing ref- erences to creative works. This paper presents a corpus of 1000 annotated documents from sources such as Wikipedia, TVTropes and WikiNews that are organized in ten partitions. Each document contains multiple gold stan- dard annotations representing various annota- tion styles. The corpus is used to evaluate a se- ries of Named Entity Linking tools in order to understand the impact of the differences in an- notation styles on the reported accuracy when processing highly ambiguous entities such as names of creative works. Relaxed annotation guidelines that include overlap styles, for in- stance, lead to better results across all tools. 1 Introduction Identifying all entities from texts and linking them to modern Knowledge Graphs (KGs) like DBpe- dia and Wikidata is the core task of Named Entity Linking (NEL) (Rosales-Méndez et al., 2018b). A particular challenge in NEL is disambiguating the entity references in the surface form (original text). For example, mentions such as NYT or NY Times or even Times could all be surface forms referenc- ing the entity The New York Times and they can then be linked to the associated entries from well- known public KGs like DBPedia and Wikidata (i.e., dbr:The_New_York_Times or wd:Q9684 1). In ad- 1dbr represents the namespace abbreviation of DBpedia resources, wd for Wikidata resources dition to entities, the relations between them, as defined in the texts themselves or extracted from a KG, can also help enlighten the narratives. All of the NEL related tasks are supported by large com- munities which have a vested interest in advancing their Knowledge Extraction (KE) capabilities. The various good results for general NEL benchmarks (e.g., F1 scores of 0.80 (Rosales-Méndez et al., 2018a)) are encouraging and suggest that the tasks tested for in these benchmarks may be solved soon. However, the main focus of most reported evalu- ations to date has been on the correct annotation of several specific types of entities like Persons (PER), Locations (LOC) or Organizations (ORG). Sometimes fine-grained typing was also allowed for locations, therefore allowing us to use types like natural locations (LOC) to depict naturally occurring locations like mountains or rivers, facil- ity (FAC) to describe buildings …

Content truncated to 500 words