Framing Named Entity Linking Error Types

Framing Named Entity Linking Error Types Adrian M.P. Bra¸soveanu*, Giuseppe Rizzo‡, Philipp Kuntschik†, Albert Weichselbraun†, Lyndon J.B. Nixon* * MODUL Technology GmbH, Am Kahlenberg 1, 1190 Vienna, Austria ‡ ISMB, Via Pier Carlo Boggio 61, 10138 Torino, Italy † Swiss Institute for Information Research, University of Applied Sciences Chur, Pulvermühlestrasse 57, 7000 Chur, Switzerland {adrian.brasoveanu,lyndon.nixon}@modul.ac.at giuseppe.rizzo@ismb.it {philipp.kuntschik,albert.weichselbraun}@htwchur.ch Abstract Named Entity Linking (NEL) and relation extraction forms the backbone of Knowledge Base Population tasks. The recent rise of large open source Knowledge Bases and the continuous focus on improving NEL performance has led to the creation of automated benchmark solutions during the last decade. The benchmarking of NEL systems offers a valuable approach to understand a NEL system’s performance quantitatively. However, an in-depth qualitative analysis that helps improving NEL methods by identifying error causes usually requires a more thorough error analysis. This paper proposes a taxonomy to frame common errors and applies this taxonomy in a survey study to assess the performance of four well-known Named Entity Linking systems on three recent gold standards. Keywords: Named Entity Linking, Linked Data Quality, Corpora, Evaluation, Error Analysis 1. Introduction A Named Entity Linking (NEL) system identifies, classi- fies and links entity mentions from a text to their Knowl- edge Base (KB) references. A NEL system can also be used for extracting factual knowledge from a text in order to use it for Knowledge Base Population (KBP). The typ- ical components of a NEL system reflect the logical steps of identifying and linking the entities: identification and classification (to a defined type like person, organization or location) of the entity mentions in a text (Named En- tity Recognition and Classification or NERC), linking to the referent KB, and clustering of the remaining unlinked entities (Ji and Nothman, 2016). Some of these steps might be performed jointly if the chosen architecture supports it (e.g., neural architectures). Numerous current architectures used for NEL systems include: (i) graph-based disambigua- tion which uses the links between the entities found in the text in order to rank the best candidates; (ii) statistical dis- ambiguation typically focused on classic machine learning algorithms or heuristics (e.g., SVM, Conditional Random Fields, etc.) or (iii) neural models (e.g., LSTM). When evaluating a system, at least three other components are needed: a dataset (usually a labeled corpus or gold stan- dard), a certain KB version (e.g., DBpedia 2016-04), and a scorer that computes measures such as accuracy, precision, recall, and F-measure. These measures compare the NEL systems quantitatively, but do not help system designers improve their approaches. If, for example, one wants to improve the precision of a sys- tem because it affects its perceived quality, the number of false positives needs to be decreased. Reducing the number of errors (false positives and false negatives) requires a thor- ough analysis of the evaluation results. A good method to do this is to use the primary error analysis results from the TAC-KBP inspired neleval1 scorer (Hachey et al., 2014), which is limited to indicating whether …

Content truncated to 500 words