Improving Named Entity Linking Corpora Quality

Proceedings of Recent Advances in Natural Language Processing, pages 1328–1337, Varna, Bulgaria, Sep 2–4, 2019. https://doi.org/10.26615/978-954-452-056-4_152 1328 Improving Named Entity Linking Corpora Quality Albert Weichselbraun1, Adrian M.P. Bras¸oveanu2, Philipp Kuntschik1, and Lyndon J.B. Nixon2 1Swiss Institute for Information Science, University of Applied Sciences, Chur, Switzerland {albert.weichselbraun,philipp.kuntschik}@htwchur.ch 2MODUL Technology GmbH, Vienna, Austria {adrian.brasoveanu,lyndon.nixon}@modul.ac.at Abstract Gold standard corpora and competitive evaluations play a key role in benchmark- ing named entity linking (NEL) perfor- mance and driving the development of more sophisticated NEL systems. The quality of the used corpora and the used evaluation metrics are crucial in this process. We, therefore, assess the qual- ity of three popular evaluation corpora, identifying four major issues which af- fect these gold standards: (i) the use of different annotation styles, (ii) incorrect and missing annotations, (iii) Knowledge Base evolution, (iv) and differences in an- notating co-occurrences. This paper ad- dresses these issues by formalizing NEL annotations and corpus versioning which allows standardizing corpus creation, sup- ports corpus evolution, and paves the way for the use of lenses to automatically trans- form between different corpus configura- tions. In addition, the use of clearly de- fined scoring rules and evaluation metrics ensures a better comparability of evalua- tion results. 1 Introduction Named Entity Linking (NEL) systems identify mentions of named entities and link them to Knowledge Bases (KBs) (Ji et al., 2017). Their evaluation heavily depends upon annotated gold standards and competitions such as TAC-KBP (Ji et al., 2017) or Open Knowledge Extraction (Nuz- zolese et al., 2016) which help drive research and advance the state of the art. Knowledge Bases, annotation guidelines and gold standards, NEL tools, as well as the evaluation systems them- selves, were found to introduce errors into NEL evaluations (Bras¸oveanu et al., 2018). The most critical issues are related to corpora quality due to wrong, partial or insufficient annotations (van Erp et al., 2016; Jha et al., 2017). Annotation guidelines used to produce a gold standard come with different rules for describing whether the an- notation systems should take into account over- lapping mentions, co-references or general con- cepts mentioned in a KB (Rosales-M´endez et al., 2018). These guidelines are domain-specific and often depend on the application and task-specific context. Semantic search, for instance, would not require co-references, but they are relevant for relation extraction tasks (Rosales-M´endez et al., 2018). The mentioned issues can be approached from various angles. General improvements such as a clear definition for NEL will affect multiple com- ponents of a NEL system. More specific sugges- tions, such as KB refinements typically affect only corpora that link entities to that particular KB. In all these cases the objective should be improving the quality and transparency of both the corpus and the evaluation processes. A serious issue with current NEL evaluations is the lack of flexibility during the evaluation pro- cess which forces NEL systems to adapt to the used evaluation corpora. For example, if a KB has more name variants (e.g., Bobby Kennedy and RFK for Robert F. …

Content truncated to 500 words