ScienceExamCER: A High-Density Fine-Grained Science-Domain Corpus for Common Entity Recognition

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 4529–4546 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 4529 ScienceExamCER: A High-Density Fine-Grained Science-Domain Corpus for Common Entity Recognition Hannah Smith, Zeyu Zhang, John Culnan, Peter Jansen School of Information, University of Arizona Tucson, Arizona, USA pajansen@email.arizona.edu Abstract Named entity recognition identifies common classes of entities in text, but these entity labels are generally sparse, limiting utility to downstream tasks. In this work we present ScienceExamCER, a densely-labeled semantic classification corpus of 133k mentions in the science exam domain where nearly all (96%) of content words have been annotated with one or more fine-grained semantic class labels including taxonomic groups, meronym groups, verb/action groups, properties and values, and synonyms. Semantic class labels are drawn from a manually-constructed fine-grained typology of 601 classes generated through a data-driven analysis of 4,239 science exam questions. We show an off-the-shelf BERT-based named entity recognition model modified for multi-label classification achieves an accuracy of 0.85 F1 on this task, suggesting strong utility for downstream tasks in science domain question answering requiring densely-labeled semantic classification. Keywords: named entity recognition, corpus, science 1. Introduction Named entity recognition (NER) (Grishman and Sundheim, 1996) is a common natural language processing task that aims to abstract or categorize common classes of noun phrases in text, such as identifying “Arthur” as a person or “Montreal” as a location. This high-level categoriza- tion of important entities in text is a staple of most modern NLP pipelines, and has a variety of applications for higher- level tasks including information extraction (Valenzuela- Esc´arcega et al., 2016), knowledge base population (Dredze et al., 2010), and question answering (Abujabal et al., 2017). Named entity recognition identifies common classes of entities in text, but these entity labels are generally sparse (typically occurring for between 10% to 20% of words in a corpus, see Section 3.4.), limiting utility to downstream tasks. In this work, we introduce the idea of common entity recognition (CER), which aims to tag all content words in text with an appropriate fine-grained semantic class. CER allows text to be automatically annotated with a much richer set of semantic labels, potentially providing greater utility for downstream applications such as question answering or automated inference. We explore CER in the context of scientific text, and present ScienceExamCER, a training corpus annotated with over 113k common entity annotations drawn from a fine-grained set of over 600 se- mantic categories, which include named entities, as well as verb groups, properties and quantities, thematic types, and synonyms for key terminology. We also release an off-the- shelf NER tagger modified to perform multilabel CER tag- ging. This BERT CER tagger achieves an accuracy of 0.85 F1 on this task, indicating that the tag ontology labels are well-defined and clearly identifiable. These two resources offer new opportunities to explore the impact of dense se- mantic annotation in downstream tasks. We believe the notion of dense fine-grained semantic tagging to be potentially useful …

Content truncated to 500 words