SemEval 2017 Task 10: ScienceIE - Extracting Keyphrases and Relations from Scientific Publications

Proceedings of the 11th International Workshop on Semantic Evaluations (SemEval-2017), pages 546–555, Vancouver, Canada, August 3 - 4, 2017. c⃝2017 Association for Computational Linguistics SemEval 2017 Task 10: ScienceIE - Extracting Keyphrases and Relations from Scientific Publications Isabelle Augenstein1, Mrinal Das2, Sebastian Riedel1, Lakshmi Vikraman2, and Andrew McCallum2 1Department of Computer Science, University College London (UCL), UK 2College of Information and Computer Sciences, University of Massachusetts Amherst, USA Abstract We describe the SemEval task of extract- ing keyphrases and relations between them from scientific documents, which is cru- cial for understanding which publications describe which processes, tasks and ma- terials. Although this was a new task, we had a total of 26 submissions across 3 eval- uation scenarios. We expect the task and the findings reported in this paper to be relevant for researchers working on under- standing scientific content, as well as the broader knowledge base population and information extraction communities. 1 Introduction Empirical research requires gaining and maintain- ing an understanding of the body of work in spe- cific area. For example, typical questions re- searchers face are which papers describe which tasks and processes, use which materials and how those relate to one another. While there are re- view papers for some areas, such information is generally difficult to obtain without reading a large number of publications. Current efforts to address this gap are search en- gines such as Google Scholar,1 Scopus2 or Seman- tic Scholar,3 which mainly focus on navigating au- thor and citations graphs. The task tackled here is mention-level iden- tification and classification of keyphrases, e.g. Keyphrase Extraction (TASK), as well as extract- ing semantic relations between keywords, e.g. Keyphrase Extraction HYPONYM-OF Informa- tion Extraction. These tasks are related to the tasks of named entity recognition, named entity 1https://scholar.google.co.uk/ 2http://www.scopus.com/ 3https://www.semanticscholar.org/ classification and relation extraction. However, keyphrases are much more challenging to identify than e.g. person names, since they vary signifi- cantly between domains, lack clear signifiers and contexts and can consist of many tokens. For this purpose, a double-annotated corpus of 500 pub- lications with mention-level annotations was pro- duced, consisting of scientific articles of the Com- puter Science, Material Sciences and Physics do- mains. Extracting keyphrases and relations between them is of great interest to scientific publishers as it helps to recommend articles to readers, high- light missing citations to authors, identify poten- tial reviewers for submissions, and analyse re- search trends over time. Note that organising keyphrases in terms of synonym and hypernym re- lations is particularly useful for search scenarios, e.g. a reader may search for articles on informa- tion extraction, and through hypernym prediction would also receive articles on named entity recog- nition or relation extraction. We expect the outcomes of the task to be rele- vant to the wider information extraction, knowl- edge base population and knowledge base con- struction communities, as it offers a novel appli- cation domain for methods researched in that area, while still offering domain-related challenges. Since the dataset is annotated for three tasks …

Content truncated to 500 words