SemEval-2018 Task 7: Semantic Relation Extraction and Classification in Scientific Papers

Proceedings of the 12th International Workshop on Semantic Evaluation (SemEval-2018), pages 679–688 New Orleans, Louisiana, June 5–6, 2018. ©2018 Association for Computational Linguistics SemEval-2018 Task 7: Semantic Relation Extraction and Classification in Scientific Papers Kata G´abor1, Davide Buscaldi1, Anne-Kathrin Schumann2, Behrang QasemiZadeh3, Ha¨ıfa Zargayouna1, Thierry Charnois1 1 LIPN, CNRS (UMR 7030), Universit´e Paris 13 firstname.lastname@lipn.univ-paris13.fr 2 ProTechnology GmbH, Dresden annek schumann@gmx.de 3 DFG SFB 991, Heinrich-Heine University, Dsseldorf zadeh@phil.uni-duesseldorf.de Abstract This paper describes the first task on seman- tic relation extraction and classification in sci- entific paper abstracts at SemEval 2018. The challenge focuses on domain-specific seman- tic relations and includes three different sub- tasks. The subtasks were designed so as to compare and quantify the effect of different pre-processing steps on the relation classifica- tion results. We expect the task to be relevant for a broad range of researchers working on extracting specialized knowledge from domain corpora, for example but not limited to sci- entific or bio-medical information extraction. The task attracted a total of 32 participants, with 158 submissions across different scenar- ios. 1 Introduction One of the emerging trends of natural language technologies is their use for the humanities and sci- ences. Recent works in the semantic web (Osborne and Motta, 2015; Wolfram, 2016) and natural lan- guage processing (Tsai et al., 2013; Luan et al., 2017; Augenstein and Søgaard, 2017; Kim et al., 2010) aimed to improve the access to scientific literature, and in particular to respond to informa- tion needs that are currently beyond the capabilities of standard search engines. Such queries include finding all papers that address a problem in a spe- cific way, or discovering the roots of a certain idea. This ambition involves the identification and classi- fication of concepts, and the relations connecting them. The purpose of the task is to automatically iden- tify relevant domain-specific semantic relations in a corpus of scientific publications. In particular, we search for and classify relations that provide snippets of information such as ”a (new) method is proposed for a task”, or ”a phenomenon is found in a certain context”, or ”results of different exper- iments are compared to each other”. Identifying such semantic relations between domain-specific concepts allows us to detect research papers which deal with the same problem, or to track the evolu- tion of results on a certain task. 2 Related Work SemEval 2010 Task 8 (Hendrickx et al., 2010) proposed a discrete classification of word pairs into 9 semantic relations, however, this task was not tailored to the needs of scientific text analysis as neither relation types nor the vocabulary were domain-specific. SemEval 2012 Task 2 (Jurgens et al., 2012) proposed a gradual notion of relational similarity: the task was to quantify the similarity between examples of relation instances. The data set was aimed at evaluating specific semantic rep- resentations for relational similarity, but does not fit our task: in this task, entity pairs were treated as static class instances; in particular, they were presented without any context. However, the re- …

Content truncated to 500 words