TDMSci: A Specialized Corpus for Scientific Literature Entity Tagging of Tasks Datasets and Metrics

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 707–714 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 707 TDMSci: A Specialized Corpus for Scientific Literature Entity Tagging of Tasks Datasets and Metrics Yufang Hou, Charles Jochim, Martin Gleize, Francesca Bonin and Debasis Ganguly IBM Research Europe, Ireland {yhou|mgleize|fbonin|debasga1}@ie.ibm.com Abstract Tasks, Datasets and Evaluation Metrics are important concepts for understanding experi- mental scientific papers. However, most previ- ous work on information extraction for scien- tific literature mainly focuses on the abstracts only, and does not treat datasets as a separate type of entity (Zadeh and Schumann, 2016; Luan et al., 2018). In this paper, we present a new corpus that contains domain expert an- notations for Task (T), Dataset (D), Metric (M) entities on 2,000 sentences extracted from NLP papers. We report experiment results on TDM extraction using a simple data augmen- tation strategy and apply our tagger to around 30,000 NLP papers from the ACL Anthology. The corpus is made publicly available to the community for fostering research on scientific publication summarization (Erera et al., 2019) and knowledge discovery. 1 Introduction The recent years have witnessed a significant growth in the number of scientific publications and benchmarks in many disciplines. As an example, in the year 2019 alone, more than 170k papers were submitted to the pre-print repository arXiv1 and among them, close to 10k papers were classified as NLP papers (i.e., cs.CL). Each empirical field of science, including NLP, will benefit from the massive increase in studies, benchmarks, and eval- uations, as they can provide ingredients for novel scientific advancements. However, researchers may struggle to keep track of all studies published in a particular field, result- ing in duplication of research, comparisons with old or outdated benchmarks, and lack of progress. In order to tackle this problem, recently there have 1https://arxiv.org/help/stats/2019_by_ area been a few manual efforts to summarize the state- of-the-art on selected subfields of NLP in the form of leaderboards that extract tasks, datasets, metrics and results from papers, such as NLP-progress2 or paperswithcode.3 But these manual efforts are not sustainable over time for all NLP tasks. Over the past few years, several studies and shared tasks have begun to tackle the task of entity extraction from scientific papers. Augenstein et al. (2017) formalized a task to identify three types of entities (i.e., task, process, material) in scien- tific publications (SemEval 2017 task10). G´abor et al. (2018) presented a task (SemEval 2018 task 7) on semantic relation extraction from NLP pa- pers. They provided a dataset of 350 abstracts and reuse the entity annotations from Zadeh and Schu- mann (2016). Recently Luan et al. (2018) released a corpus containing 500 abstracts with six types of entity annotations. However, these corpora do not treat Dataset as a separate type of entity and most of them focus on the abstracts only. In a previous study, we developed an IE sys- tem to extract {task, dataset, metric} triples from NLP …

Content truncated to 500 words