MS-Mentions: Consistently Annotating Entity Mentions in Materials Science Procedural Text

MS-MENTIONS: Consistently Annotating Entity Mentions in Materials Science Procedural Text Tim O’Gorman3∗, Zach Jensen2, Sheshera Mysore1, Rubayyat Mahbub2, Kevin Huang2, Elsa Olivetti2, Andrew McCallum1 1UMass Amherst 2MIT 3Thorn {smysore,mccallum}.cs.umass.edu {zjensen,kjhuang,rmahbub,elsao}@mit.edu Abstract Material science synthesis procedures are a promising domain for scientific NLP, as proper modeling of these recipes could provide in- sight into new ways of creating materials. However, a fundamental challenge in build- ing information extraction models for mate- rial science synthesis procedures is getting accurate labels for the materials, operations, and other entities of those procedures. We present a new corpus of entity mention an- notations over 595 Material Science synthe- sis procedural texts (157,488 tokens), which greatly expands the training data available for the Named Entity Recognition task. We out- line a new label inventory designed to pro- vide consistent annotations and a new annota- tion approach intended to maximize the con- sistency and annotation speed of domain ex- perts. Inter-annotator agreement studies and baseline models trained upon the data suggest that the corpus provides high-quality annota- tions of these mention types. This corpus helps lay a foundation for future high-quality model- ing of synthesis procedures. 1 Introduction The Material Science literature contains millions of synthesis procedures: descriptions that outline the specific steps required to create a particular material, such as the text in Figure 1. Large-scale analysis of these procedures could enable tasks such as automatic planning of new synthesis pro- cedures (Kim et al., 2020). However, such tasks require extraction of high-quality representations of those synthesis procedures from raw text (Kim et al., 2017), and thus are limited by the size and quality of annotations. An important component for information extrac- tion of synthesis procedures involves predicting the entities, actions and attributes of the synthe- sis, including the steps followed in the synthesis ∗* Work done while at UMass Amherst Figure 1: Part of an example synthesis procedure in- cluded in the dataset with entity annotations from Zhao et al. (2015). Colors represent entity types and un- derlines represent span boundaries. Colors: Target, Nonrecipe-operation, Unspecified-Material, Opera- tion, Material, Condition-Unit, Number. procedures, the materials used and created in the synthesis, and all the quantitative attributes (includ- ing operation conditions, material amounts, and properties) necessary to replicate or understand the procedure. Such annotations require the judgment of domain experts, as not all verbs in a text are actual steps in the synthesis, and not all materi- als mentioned are actual inputs or outputs of the synthesis process. We outline an inventory of 15 mention types which can be consistently annotated while still making important distinctions regard- ing the roles that these operations and mentions play in the overall synthesis. Figure 1 illustrates how such an example synthesis procedure would be annotated with entity mentions. We annotate these labels over a new corpus of 595 synthesis procedures. To improve the consis- tency of this annotation pipeline, we separate the annotation of operation-type mentions, material- type mentions, and quantitative mentions (includ- ing conditions, amounts and properties), allowing annotators to …

Content truncated to 500 words