CiteWorth: Cite-Worthiness Detection for Improved Scientific Document Understanding

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1796–1807 August 1–6, 2021. ©2021 Association for Computational Linguistics 1796 CITEWORTH: Cite-Worthiness Detection for Improved Scientific Document Understanding Dustin Wright and Isabelle Augenstein Dept. of Computer Science University of Copenhagen Denmark {dw|augenstein}@di.ku.dk Abstract Scientific document understanding is challeng- ing as the data is highly domain specific and diverse. However, datasets for tasks with scien- tific text require expensive manual annotation and tend to be small and limited to only one or a few fields. At the same time, scientific docu- ments contain many potential training signals, such as citations, which can be used to build large labelled datasets. Given this, we present an in-depth study of cite-worthiness detection in English, where a sentence is labelled for whether or not it cites an external source. To accomplish this, we introduce CITEWORTH, a large, contextualized, rigorously cleaned la- belled dataset for cite-worthiness detection built from a massive corpus of extracted plain- text scientific documents. We show that CITEWORTH is high-quality, challenging, and suitable for studying problems such as do- main adaptation. Our best performing cite- worthiness detection model is a paragraph- level contextualized sentence labelling model based on Longformer, exhibiting a 5 F1 point improvement over SciBERT which considers only individual sentences. Finally, we demon- strate that language model fine-tuning with cite-worthiness as a secondary task leads to im- proved performance on downstream scientific document understanding tasks. 1 Introduction Building effective NLP systems from scientific text is challenging due to the highly domain-specific and diverse nature of scientific language, and a lack of abundant sources of labelled data to capture this. While large scale repositories of extracted, structured, and unlabelled plain-text scientific doc- uments have recently been introduced (Lo et al., 2020), most datasets for downstream tasks such as named entity recognition (Li et al., 2016) and citation intent classification (Cohan et al., 2019) remain limited in size and highly domain specific. This begs the question: what useful training sig- nals can be automatically extracted from massive unlabelled scientific text corpora to help improve systems for scientific document processing? Scientific documents contain much inherent structure (sections, tables, equations, citations, etc.), which can facilitate creating large labelled datasets. Some recent examples include using pa- per field (Beltagy et al., 2019), the section to which a sentence belongs (Cohan et al., 2019), and the cite-worthiness of a sentence (Cohan et al., 2019; Sugiyama et al., 2010) as a training signal. Cite-worthiness detection is the task of identify- ing citing sentences, i.e. sentences which contain a reference to an external source. It has useful appli- cations, such as in assistive document editing, and as a first step in citation recommendation (Färber et al., 2018b). In addition, cite-worthiness has been shown to be useful in helping to improve the ability of models to learn other tasks (Cohan et al., 2019). We also hypothesize that there is a strong domain shift between how different fields use citations, and that such a dataset is useful for studying domain …

Content truncated to 500 words