Practical, Efficient, and Customizable Active Learning for Named Entity Recognition in the Digital Humanities

Proceedings of NAACL-HLT 2019, pages 2223–2234 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 2223 Practical, Efficient, and Customizable Active Learning for Named Entity Recognition in the Digital Humanities Alexander Erdmann,†‡ David Joseph Wrisley,‡ Benjamin Allen,† Christopher Brown,† Sophie Cohen-Bod´en`es,∗ Micha Elsner,† Yukun Feng,† Brian Joseph,† B´eatrice Joyeux-Prunel∗ and Marie-Catherine de Marneffe† †Ohio State University, USA ‡New York University Abu Dhabi, UAE ∗Ecole Normale Sup´erieure, France {ae1541,djw12}@nyu.edu,{allen.2021,brown.2583,elsner.14, feng.749,joseph.1,demarneffe.1}@osu.edu, sophiebodenes@gmail.com, beatrice.joyeux-prunel@ens.fr Abstract Scholars in inter-disciplinary fields like the Digital Humanities are increasingly interested in semantic annotation of specialized corpora. Yet, under-resourced languages, imperfect or noisily structured data, and user-specific clas- sification tasks make it difficult to meet their needs using off-the-shelf models. Manual an- notation of large corpora from scratch, mean- while, can be prohibitively expensive. Thus, we propose an active learning solution for named entity recognition, attempting to maxi- mize a custom model’s improvement per addi- tional unit of manual annotation. Our system robustly handles any domain or user-defined label set and requires no external resources, enabling quality named entity recognition for Humanities corpora where such resources are not available. Evaluating on typologically dis- parate languages and datasets, we reduce re- quired annotation by 20-60% and greatly out- perform a competitive active learning baseline. 1 Introduction Reaping the benefits of recent advances in Named Entity Recognition (NER) is challenging when dealing with under-resourced languages, niche do- mains, imperfect or noisily structured data, or user-specific classification tasks. Scholars in inter- disciplinary fields like the Digital Humanities (DH) are increasingly interested in semantic an- notation of specialized corpora that invoke many of these challenges. Thus, such corpora cannot easily be annotated automatically with blackbox, off-the-shelf NER models. Manual annotation of large corpora from scratch, meanwhile, can be prohibitively costly. Successful DH initiatives like the Pelagios Commons (Simon et al., 2016), which collects geospatial data from historical sources, often require extensive funding, relying on con- siderable manual annotation (Simon et al., 2017). To this end, we introduce the Humanities Entity Recognizer (HER),1 a whitebox toolkit for build- your-own NER models, freely available for public use. HER robustly handles any domain and user- defined label set, guiding users through an active learning process whereby sentences are chosen for manual annotation that are maximally informative to the model. Informativeness is determined based on novel interpretations of the uncertainty, repre- sentativeness, and diversity criteria proposed by Shen et al. (2004). In contrast to literature em- phasizing the disproportionate or exclusive impor- tance of uncertainty (Shen et al., 2017; Zhu et al., 2008; Olsson, 2009), we observe significant im- provements by integrating all three criteria. In addition to a robust active learning based NER toolkit, we also contribute a novel evalua- tion framework. This inclusive framework con- siders the accuracy with which an entire corpus is annotated, regardless of which instances are an- notated manually versus automatically, such that no instance is held out when the active learn- ing algorithm considers candidates for annotation. The standard, exclusive evaluation framework, by contrast, only …

Content truncated to 500 words