Comparative evaluation of boundary-relaxed annotation for Entity Linking performance

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 8238–8253 July 9-14, 2023 ©2023 Association for Computational Linguistics Comparative evaluation of boundary-relaxed annotation for Entity Linking performance Gabriel Herman Bernardim Andrade Shuntaro Yada Eiji Aramaki Graduate School of Information Science Nara Institute of Science and Technology Ikoma, Nara, Japan {herman_bernardim_andrade.hi1, s-yada, aramaki}@is.naist.jp Abstract Entity Linking performance has a strong re- liance on having a large quantity of high-quality annotated training data available. Yet, man- ual annotation of named entities, especially their boundaries, is ambiguous, error-prone, and raises many inconsistencies between an- notators. While imprecise boundary annotation can degrade a model’s performance, there are applications where accurate extraction of enti- ties’ surface form is not necessary. For those cases, a lenient annotation guideline could re- lieve the annotators’ workload and speed up the process. This paper presents a case study designed to verify the feasibility of such an- notation process and evaluate the impact of boundary-relaxed annotation in an Entity Link- ing pipeline. We first generate a set of noisy ver- sions of the widely used AIDA CoNLL-YAGO dataset by expanding the boundaries subsets of annotated entity mentions and then train three Entity Linking models on this data and evaluate the relative impact of imprecise annotation on entity recognition and disambiguation perfor- mances. We demonstrate that the magnitude of effects caused by noise in the Named Entity Recognition phase is dependent on both model complexity and noise ratio, while Entity Disam- biguation components are susceptible to entity boundary imprecision due to strong vocabulary dependency. 1 Introduction Out of many tasks under the Natural Language Processing (NLP) umbrella, Entity Linking (EL) is one of the critical steps for Information Extrac- tion, allowing the retrieval and understanding of information from unstructured textual sources. By definition, EL is the task of associating a Named Entity (NE) mention (a concept, person, location, etc.) to a unique identifier that describes what that mention refers to in a knowledge base (KB) (Zhang et al., 2021). This is especially significant in the biomedical domain, where it can be used to asso- ciate mentions with different surface forms to a single concept in an ontology, such as the Unified Medical Language System (UMLS)1, allowing for better indexing and relation extraction (Leaman et al., 2015). In this paper’s definition, EL comprises two phases, as presented in Figure 1. Named Entity Recognition (NER) searches the sentence for NE mentions, and then Entity Disambiguation (ED) assigns a unique identifier for each one. (Kolitsas et al., 2018; Özge Sevgili et al., 2022). NER is also commonly related to classifying a detected mention into a set of categories. However, for this study, we consider it as only the location of a NE in the text. Although some methodologies have presented state-of-the-art (SoTA) results for this task (De Cao et al., 2021; Yamada et al., 2020), large amounts of high-quality labeled data are still crucial to achieve good performance models. Yet, text annotation is no trivial task. …

Content truncated to 500 words