Comparing Annotated Datasets for Named Entity Recognition in English Literature

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 3788–3797 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 3788 Comparing Annotated Datasets for Named Entity Recognition in English Literature Rositsa V. Ivanova,1 Sabrina Kirrane,1 and Marieke van Erp2 1Vienna University of Economics and Business Welthandelsplatz 1, 1020, Vienna, Austria 2KNAW Humanities Cluster Oudezijds Achterburgwal 185, 1012 DK, Amsterdam, The Netherlands {rivanova,skirrane}@wu.ac.at, marieke.van.erp@dh.huc.knaw.nl Abstract The growing interest in named entity recognition (NER) in various domains has led to the creation of different benchmark datasets, often with slightly different annotation guidelines. To better understand the different NER benchmark datasets for the domain of English literature and their impact on the evaluation of NER tools, we analyse two existing annotated datasets and create two additional gold standard datasets. Following on from this, we evaluate the performance of two NER tools, one domain-specific and one general-purpose NER tool, using the four gold standards, and analyse the sources for the differences in the measured performance. Our results show that the performance of the two tools varies significantly depending on the gold standard used for the individual evaluations. Keywords: named entity recognition, corpus annotation, annotated datasets, annotation guidelines, literature 1. Introduction Over the years, researchers and engineers have inves- tigated various approaches that could potentially im- prove the recognition and linking of named entities. These different approaches include using deep learning to extract entities (Li et al., 2020), providing tools with more annotated training data, for instance by making use of technologies such as linked databases to link en- tities despite diverse name variations (Zhu and Iglesias, 2018), enriching gazetteers (i.e. lists of aliases of en- tities) and using dictionaries (i.e. entity mappings) for better linking between entities (Hulpus et al., 2013). Most of these approaches have indeed led to signifi- cant improvements in the performance of named entity recognition (NER) tools. However, as we showcase in the following examples from the English literature do- main, there is still room for improvement. Generally speaking, the majority of NER tools strug- gle to perform well when the entities in the text con- tain specific characteristics. When it comes to English novels Dekker et al. (2019) observe the poor perfor- mance of off-the-shelf tools when names contain char- acters, which are used in an unusual manner for that particular language (e.g. d’Artagnan). In historical let- ters, Mac Kim and Cassidy (2015) describe the lack of cues to differentiate between names and ordinary words as one of the biggest challenges when it comes to automated analysis. Furthermore, Woldenga-Racine (2019) conclude that the most frequent cause for incor- rect NER in old English novels is the use of capitalized words for non-named entities. Although, many tools rely on gazetteers and rules for NER, the gazetteers and machine learning methods used to enrich them are mostly based on modern English, making the analysis of old documents, including books, difficult and less accurate. In this paper, we take a closer look at existing …

Content truncated to 500 words