An annotated dataset of literary entities

Proceedings of NAACL-HLT 2019, pages 2138–2144 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 2138 An Annotated Dataset of Literary Entities David Bamman School of Information UC Berkeley dbamman@berkeley.edu Sejal Popat School of Information UC Berkeley sejal@berkeley.edu Sheng Shen Computer Science Division UC Berkeley sheng.s@berkeley.edu Abstract We present a new dataset comprised of 210,532 tokens evenly drawn from 100 different English- language literary texts annotated for ACE entity categories (person, location, geo-political entity, facility, organization, and vehicle). These cate- gories include non-named entities (such as “the boy”, “the kitchen”) and nested structure (such as [[the cook]’s sister]). In contrast to existing datasets built primarily on news (focused on geo- political entities and organizations), literary texts offer strikingly different distributions of entity cat- egories, with much stronger emphasis on people and description of settings. We present empirical results demonstrating the performance of nested entity recognition models in this domain; training natively on in-domain literary data yields an im- provement of over 20 absolute points in F-score (from 45.7 to 68.3), and mitigates a disparate im- pact in performance for male and female entities present in models trained on news data. 1 Introduction Computational literary analysis works at the inter- section of natural language processing and literary studies, drawing on the structured representation of text to answer literary questions about character (Underwood et al., 2018), objects (Tenen, 2018) and place (Evans and Wilkens, 2018). Much of this work relies on the ability to extract entities accurately, including work focused on modeling (Bamman et al., 2014; Iyyer et al., 2016; Chaturvedi et al., 2017). And yet, with notable ex- ceptions (Vala et al., 2015; Brooke et al., 2016), nearly all of this work tends to use NER models that have been trained on non-literary data, for the simple reason that labeled data exists for domains like news through standard datasets like ACE (Walker et al., 2006), CoNLL (Tjong Kim Sang and De Meulder, 2003) and OntoNotes (Hovy et al., 2006)—and even historical non-fiction (De- Lozier et al., 2016; Rayson et al., 2017)—but not for literary texts. This is naturally problematic for several rea- sons: models trained on out-of-domain data surely degrade in performance when applied to a very different domain, and especially for NER, as Au- genstein et al. (2017) has shown; and without in- domain test data, it is difficult to directly esti- mate the severity of this degradation. At the same time, literary texts also demand slightly different representations of entities. While classic NER models typically presume a flat entity structure (Finkel and Manning, 2009), relevant characters and places (and other entities) in literature need not be flat, and need not be named: The cook’s sister ate lunch contains two PER entities ([The cook] and [The cook’s sister]). We present in this work a new dataset of en- tity annotations for a wide sample of 210,532 to- kens from 100 literary texts to help address these issues and help advance computational work on literature. These annotations …

Content truncated to 500 words