Comparative Analysis of Named Entity Recognition in the Dungeons and Dragons Domain

Proceedings of Recent Advances in Natural Language Processing, pages 1225–1233 Varna, Sep 4–6, 2023 https://doi.org/10.26615/978-954-452-092-2_130 1225 Comparative Analysis of Named Entity Recognition in the Dungeons and Dragons Domain Gayashan Weerasundara and Nisansa de Silva Department of Computer Science & Engineering University of Moratuwa, Sri Lanka {gayashan.22, NisansaDdS}@cse.mrt.ac.lk Abstract Many NLP tasks, although well-resolved for general English, face challenges in specific do- mains like fantasy literature. This is evident in Named Entity Recognition (NER), which detects and categorizes entities in text. We ana- lyzed 10 NER models on 7 Dungeons and Drag- ons (D&D) adventure books to assess domain- specific performance. Using open-source Large Language Models, we annotated named enti- ties in these books and evaluated each model’s precision. Our findings indicate that, without modifications, Flair, Trankit, and Spacy outper- form others in identifying named entities in the D&D context. 1 Introduction Named Entity Recognition (NER) targets the iden- tification and classification of textual entities, such as names and locations. In the diverse and in- tricate vocabulary of fantasy literature, like that of Dungeons and Dragons (D&D), NER becomes challenging (Zagal and Deterding, 2018). D&D, a prominent fantasy literature domain, spans content for its namesake tabletop game (Peiris and de Silva, 2022, 2023; Zhou et al., 2022). These narratives inhabit fictional realms like Forgotten Realms and Dragonlance, bursting with characters, locations, and objects (Gygax and Arneson, 1974). NER’s utility in fantasy literature is vast: from extracting information and summarizing text to character analysis and plot creation. How- ever, conventional NER models, primarily trained on standard datasets like CoNLL-2003 (Tjong Kim Sang and De Meulder, 2003) or OntoNotes 5.0 (Weischedel et al., 2013), might falter on fan- tasy texts due to their unique linguistic attributes. Recognizing the need for domain-specific adapta- tion, other specialized areas such as law (Sugath- adasa et al., 2017), medicine (de Silva et al., 2017), and the dynamic landscape of social me- dia (de Silva and Dou, 2021) have already seen research emphasizing it. Large models, as Yao et al. (2021) points out, can face domain adaptation challenges, stressing the need for evaluating NER models specifically on fantasy content. Fantasy NER has potential, especially with ad- vancements in image generation. A notable appli- cation might involve an image generation model leveraging NER tags to derive prompts and subse- quently produce contextually relevant images. Our study contrasts 10 NER models across seven D&D books, each averaging 118,000 words. Man- ual annotations of entities were made and juxta- posed against model outputs. Through precision assessments and named entity distribution analyses, we glean insights into model performances in the fantasy domain. Our key contributions include: • A pioneering, comprehensive NER model evaluation on fantasy content. • An annotated D&D book dataset for NER studies. • A deep dive into varied NER models’ strengths and pitfalls in the fantasy realm. • Discussions on NER’s role and prospects in fantasy literature. Following this, Section 2 delves into related NER and fantasy literature works. Section 3 details our data and annotation process, while Section …

Content truncated to 500 words