Beyond Citations: Corpus-based Methods for Detecting the Impact of Research Outcomes on Society

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 6777–6785 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 6777 Beyond Citations: Corpus-based Methods for Detecting the Impact of Research Outcomes on Society Rezvaneh Rezapour§, Jutta Bopp†, Norman Fiedler†, Diana Steffen†, Andreas Witt†, Jana Diesner§ §University of Illinois at Urbana–Champaign, USA {rezapou2,jdiesner}@illinois.edu †Leibniz-Institut für Deutsche Sprache, Germany {bopp,fiedler,steffen,witt}@ids-mannheim.de Abstract This paper proposes, implements and evaluates a novel, corpus-based approach for identifying categories indicative of the impact of research via a deductive (top-down, from theory to data) and an inductive (bottom-up, from data to theory) approach. The resulting categorization schemes differ in substance. Research outcomes are typically assessed by using bibliometric methods, such as citation counts and patterns, or alternative metrics, such as references to research in the media. Shortcomings with these methods are their inability to identify impact of research beyond academia (bibliometrics) and considering text-based impact indicators beyond those that capture attention (altmetrics). We address these limitations by leveraging a mixed-methods approach for eliciting impact categories from experts, project personnel (deductive) and texts (inductive). Using these categories, we label a corpus of project reports per category schema, and apply supervised machine learning to infer these categories from project reports. The classification results show that we can predict deductively and inductively derived impact categories with 76.39% and 78.81% accuracy (F1-score), respectively. Our approach can complement solutions from bibliometrics and scientometrics for assessing the impact of research and studying the scope and types of advancements transferred from academia to society. Keywords: impact assessment, natural language processing, machine learning, corpus analysis, deductive and inductive category detection 1. Introduction National and independent organizations and foundations have been providing researchers across domains with op- portunities in the form of funding and infrastructure to ad- vance knowledge and discovery. The premise with research funding is that the return of investment, i.e., research out- comes, will benefit people, the economy, or the environ- ment, among other beneficiaries (Bornmann and Daniel, 2005). How do we know if this goal has been achieved? Traditionally, the performance of researchers and the im- pact of research outputs have been assessed by capturing and analyzing citations of publications. The field of infor- metrics has been developing various such metrics, which can be factors considered when allocating research funding or evaluating research (Aksnes et al., 2019). In addition to that, funders, scholars and society have become interested in measuring traceable effects of research outcomes beyond their reach within academia1 (Parker and Van Teijlingen, 2012). In response to these needs and given the growth of research being mentioned on online platforms and social media, ad- ditional metrics that capture attention to scholarly work have been developed2 (Priem et al., 2012). These metrics are being used to supplement citation counts and expanding the types of impact of research that are considered for as- sessments (Piwowar, 2013; Bornmann, 2015; Pulido et al., 2018; Subramanyam, 1983; Swanson et al., 2006; Smal- heiser and Torvik, 2008). Also, the field …

Content truncated to 500 words