SemEval-2021 Task 11: NLPContributionGraph - Structuring Scholarly NLP Contributions for a Research Knowledge Graph

Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 364–376 Bangkok, Thailand (online), August 5–6, 2021. ©2021 Association for Computational Linguistics 364 SemEval-2021 Task 11: NLPCONTRIBUTIONGRAPH - Structuring Scholarly NLP Contributions for a Research Knowledge Graph Jennifer D’Souza1, S¨oren Auer1, and Ted Pedersen2 1TIB Leibniz Information Centre for Science and Technology, Hannover, Germany {jennifer.dsouza, auer}@tib.eu 2 University of Minnesota, Duluth, USA tpederse@d.umn.edu Abstract There is currently a gap between the natural language expression of scholarly publications and their structured semantic content model- ing to enable intelligent content search. With the volume of research growing exponentially every year, a search feature operating over semantically structured content is compelling. The SemEval-2021 Shared Task NLPCONTRI- BUTIONGRAPH (a.k.a. ‘the NCG task’) tasks participants to develop automated systems that structure contributions from NLP scholarly ar- ticles in the English language. Being the first- of-its-kind in the SemEval series, the task re- leased structured data from NLP scholarly ar- ticles at three levels of information granular- ity, i.e. at sentence-level, phrase-level, and phrases organized as triples toward Knowl- edge Graph (KG) building. The sentence- level annotations comprised the few sentences about the article’s contribution. The phrase- level annotations were scientific term and pred- icate phrases from the contribution sentences. Finally, the triples constituted the research overview KG. For the Shared Task, partici- pating systems were then expected to auto- matically classify contribution sentences, ex- tract scientific terms and relations from the sen- tences, and organize them as KG triples. Overall, the task drew a strong participation de- mographic of seven teams and 27 participants. The best end-to-end task system classified con- tribution sentences at 57.27% F1, phrases at 46.41% F1, and triples at 22.28% F1. While the absolute performance to generate triples re- mains low, in the conclusion of this article, the difficulty of producing such data and as a con- sequence of modeling it is highlighted. 1 Introduction Traditional search models over scholarly communi- cation are now changing toward Knowledge Graph (KG) models operating on structured fine-grained scholarly content offering enhanced contextual search results. Several initiatives exist to this end: Google Scholar, Web of Science (Birkle et al., 2020), Microsoft Academic Graph (Wang et al., 2020), OpenAIRE Research Graph (Manghi et al., 2019), Open Research Knowledge Graph (Auer, 2018), Semantic Scholar (Fricke, 2018) to name just a few. These KG models differ in their content, their level of detail, etc., as they represent diverse aspects of scholarly communication. Text, of course, is of seminal importance to Sci- ence. It is as important as experimentation itself; unpublished research lacks validity. Seen in an- other angle, it is hard to imagine a medium other than discourse that can convey a comprehensive picture of the scholarly investigation. For the wider research audience, it is interesting to read the full “stories” of Science. Nonetheless, since scientific literature is grow- ing at a rapid rate (Johnson et al., 2018) and re- searchers today are faced with this publications deluge (Landhuis, 2016), it is increasingly tedious, if not practically …

Content truncated to 500 words