SciREX: A Challenge Dataset for Document-Level Information Extraction

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7506–7516 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 7506 SCIREX: A Challenge Dataset for Document-Level Information Extraction Sarthak Jain2∗ Madeleine van Zuylen1 Hannaneh Hajishirzi1,3 Iz Beltagy1 Allen Institute for AI1 Northeastern University2 University of Washington3 jain.sar@northeastern.edu {madeleinev,hannah,beltagy}@allenai.org Abstract Extracting information from full documents is an important problem in many domains, but most previous work focus on identifying re- lationships within a sentence or a paragraph. It is challenging to create a large-scale in- formation extraction (IE) dataset at the doc- ument level since it requires an understand- ing of the whole document to annotate enti- ties and their document-level relationships that usually span beyond sentences or even sec- tions. In this paper, we introduce SCIREX, a document level IE dataset that encompasses multiple IE tasks, including salient entity iden- tification and document level N-ary relation identification from scientific articles. We an- notate our dataset by integrating automatic and human annotations, leveraging existing scien- tific knowledge resources. We develop a neu- ral model as a strong baseline that extends pre- vious state-of-the-art IE models to document- level IE. Analyzing the model performance shows a significant gap between human per- formance and current baselines, inviting the community to use our dataset as a challenge to develop document-level IE models. Our data and code are publicly available at https: //github.com/allenai/SciREX 1 Introduction Extracting information about entities and their re- lationships from unstructured text is an impor- tant problem in NLP. Conventional datasets and methods for information extraction (IE) focus on within-sentence relations from general Newswire text (Zhang et al., 2017). However, recent work started studying the development of full IE models and datasets for short paragraphs (e.g., information extraction from abstracts of scientific articles as in SCIERC (Luan et al., 2018)), or only extracting ∗Work done while at AI2 We evaluate our model on the task of question answering using Section : Dataset SQuAD is a machine comprehension dataset on a large set of Wikipedia articles , ….. . Two metrics are used to evaluate models : Exact Match ( EM ) and a softer metric , F1 score ….. . Section: Model Details . … Each paragraph and question are tokenized by a regular - expression - based word tokenizer ( PTB Tokenizer ) and fed into the model . …. Section : Results . The results of our model and competing approaches on the hidden test are summarized in Table [ reference ] . BiDAF ( ensemble ) achieves an EM score of 73.3 and an F1 score of 81.1 , outperforming all previous approaches . Figure 1: An example showing annotations for entity mentions ( Dataset , Metric , Task , Method ), coref- erences (indicated by arrows), salient entities (bold), and N-ary relation (SQuaD, Machine Comprehension, BiDAF (ensemble), EM/F1) that can only be extracted by aggregating information across sections. relations (given ground truth entities) on long doc- uments (e.g. Jia et al. (2019)). …

Content truncated to 500 words