Scientific Statement Classification over arXiv.org

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 1219–1226 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 1219 Scientific Statement Classification over arXiv.org Deyan Ginev, Bruce R. Miller FAU Erlangen-Nuremberg, National Institute of Standards and Technology dginev@kwarc.info, bruce.miller@nist.gov Abstract We introduce a new classification task for scientific statements and release a large-scale dataset for supervised learning. Our resource is derived from a machine-readable representation of the arXiv.org collection of preprint articles. We explore fifty author-annotated categories and empirically motivate a task design of grouping 10.5 million annotated paragraphs into thirteen classes. We demonstrate that the task setup aligns with known success rates from the state of the art, peaking at a 0.91 F1-score via a BiLSTM encoder-decoder model. Additionally, we introduce a lexeme serialization for mathematical formulas, and observe that context-aware models could improve when also trained on the symbolic modality. Finally, we discuss the limitations of both data and task design, and outline potential directions towards increasingly complex models of scientific discourse, beyond isolated statements. Keywords: statement classification task, large corpora, math-rich natural language processing 1. Introduction Scientific discourse is a promising avenue for natural lan- guage processing (NLP). Scholarly works are rich in refer- ential meaning due to their conceptual focus and structured expositions. They present a multitude of targets for the de- velopment of semantic enrichment and data mining tech- niques. We survey a prime example of an openly available library of scientific texts – the arXiv.org preprint server. It is one of the largest international repositories of STEM sci- entific articles, numbering over 1.5 million submissions at the time of writing. Crucially, these texts are prepared for human academic consumption via print. It is only a re- cent development that they have been made available in a fully machine-readable representation, as part of a decade- long research endeavor (Stamerjohanns et al., 2010). The arXMLiv project now publishes an HTML5 dataset (Ginev, 2018) of 1.2 million documents converted from the original submissions – allowing for straightforward reuse in main- stream NLP pipelines. This dataset surpasses 11 billion to- kens and is sufficiently large to bootstrap pre-training lan- guage models. In this paper we outline and motivate a new statement clas- sification task, the first to be extracted from this corpus. Our goal is to fully leverage the annotations authors de- posit while visually highlighting the key statements in their texts. We have attempted to collect the full spectrum of annotated statements, ranging from standard pieces of nar- rative (e.g. abstract, related work) to specialized parts of a scientific exposition (e.g. method, result). Our emphasis is on constructing the biggest possible resource for supervised learning, while also maintaining the highest possible qual- ity in data collection. Our goal is to set the stage for further research, as well as to provide open and reproducible in- frastructure for the wider community. In Section 2, we explain the precautions needed to reliably work with the data. Next, in Section 3 …

Content truncated to 500 words