SemEval-2021 Task 8: MeasEval – Extracting Counts and Measurements and their Related Contexts

Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 306–316 Bangkok, Thailand (online), August 5–6, 2021. ©2021 Association for Computational Linguistics 306 SemEval-2021 Task 8: MeasEval – Extracting Counts and Measurements and their Related Contexts Corey A Harper1,2, Jessica Cox1, Curt Kohler1, Antony Scerri1, Ron Daniel, Jr.1 and Paul Groth2 1Elsevier Labs, Suite 800, 230 Park Avenue, New York, NY 10169, USA 1{c.harper, j.cox, c.kohler, a.scerri, r.daniel}@elsevier.com 2University of Amsterdam, Postbus 94323 / 1090 GH, Amsterdam 2{c.a.harper, p.t.groth}@uva.nl Abstract We describe MeasEval, a SemEval task of extracting counts, measurements, and related context from scientific documents, which is of significant importance to the creation of Knowledge Graphs that distill information from the scientific literature. This is a new task in 2021, for which over 75 submissions from 25 participants were received. We ex- pect the data developed for this task and the findings reported to be valuable to the scien- tific knowledge extraction, metrology, and au- tomated knowledge base construction commu- nities. 1 Introduction Counts and measurements are an important part of scientific discourse (Rijgersberg et al., 2011). It is relatively easy to find measurements in text (Fop- piano et al., 2019a), but a bare measurement like 17mg is not informative without knowing what it is referring to. For example, it is important to know whether a quantity is 17mg of a medicine dosage or 17mg of concrete additive. Only recently have at- tempts been made to identify the named entity and property being measured (Hundman and Maamann, 2017). Extracting such information is challenging because the way scientists write can be ambiguous and inconsistent. Furthermore, the location of this information relative to the measurement can vary greatly, and might even be in a different sentence. Being able to extract measurement informa- tion automatically can enable the construction of databases of measured properties. Such databases are important in biomedicine (Hao et al., 2016), en- gineering (Foppiano et al., 2019a), and other scien- tific disciplines (Bergmann et al., 2017), but the ap- proaches used for populating these databases do not generalize widely. Furthermore, knowledge graphs (Hogan et al., 2021) frequently aggregate quanti- tative data reported in the literature and are often built through a largely manual curation process. Ex- amples include: LITTERBASE1 (Bergmann et al., 2017), which aggregates observations of marine litter distribution; NeuroElectro2 (Tripathy et al., 2014), which collects information on electrophysi- ological properties of neurons; and various model organism databases like the Zebrafish Information Network3 (Sprague, 2006), which provide sum- maries of gene information. Beyond knowledge graphs and curated databases, clinical health contexts often require extraction of measured values for lab results and patient observations. Moreover, scientific research frequently relies on precise measurements for reproducibility of experimental methods (Kaiser, 2018). Measured property extraction could be of value in many other contexts, such as fact checking and news validation or in statistical analysis for public policy (Einav and Levin, 2014). Research in information extraction and knowl- edge graph creation has concentrated on forming triples by extracting entities and relations (Kon- …

Content truncated to 500 words