Evaluating Scholarly Impact: Towards Content-Aware Bibliometrics

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6041–6053 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 6041 Evaluating Scholarly Impact: Towards Content-Aware Bibliometrics Saurav Manchanda and George Karypis University of Minnesota Twin Cities, MN, USA manch043,karypis@umn.edu Abstract Quantitatively measuring the impact-related aspects of scientific, engineering, and tech- nological (SET) innovations is a fundamen- tal problem with broad applications. Tradi- tional citation-based measures for assessing the impact of innovations and related entities do not take into account the content of the publications. This limits their ability to pro- vide rigorous quality-related metrics because they cannot account for the reasons that led to a citation. We present approaches to esti- mate content-aware bibliometrics to quantita- tively measure the scholarly impact of a pub- lication. Our approaches assess the impact of a cited publication by the extent to which the cited publication informs the citing publi- cation. We introduce a new metric, called Con- tent Informed Index (CII), that uses the content of the paper as a source of distant-supervision, to quantify how much the cited-node informs the citing-node. We evaluate the weights esti- mated by our approach on three manually an- notated datasets, where the annotations quan- tify the extent of information in the citation. Particularly, we evaluate how well the ranking imposed by our approach associates with the ranking imposed by the manual annotations. CII achieves up to 103% improvement in per- formance as compared to the second-best per- forming approach. 1 Introduction Scientific, engineering, and technological (SET) innovations have been the drivers behind many of the significant positive advances in our modern economy, society, and life. To measure various impact-related aspects of these innovations various quantitative metrics have been developed and de- ployed. These metrics play an important role as they are used to influence how resources are allo- cated, assess the performance of personnel, identify intellectual property (IP)-related takeover targets, value a company’s intangible assets, and identify strategic and/or emerging competitors. Citation networks of peered-reviewed scholarly publications (e.g., journal/conference articles and patents) have widely been used and studied in order to derive such metrics for the various entities in- volved (e.g., articles, researchers, institutions, com- panies, journals, conferences, countries, etc. (Agui- nis et al., 2012)). However, most of these tradi- tional metrics, such as citation counts and h-index, treat all citations and publications equally and do not take into account the content of the publications and the context in which a prior scholarly work was cited. Another related line of work, such as PageR- ank (Page et al., 1999) and HITS (Kleinberg, 1999) considers the node centrality (as a proxy for influ- ence) but still operate in a content-agnostic manner. Content-agnostic metrics fail to precisely size up the scholarly impact of an article as they do not differentiate between the possible reasons that a scholarly work is being cited. In addition, they can be easily manipulated by the presence of malicious entities, such as publication venues indulging in self-citations, which leads to high …

Content truncated to 500 words