NLPeer: A Unified Resource for the Computational Study of Peer Review

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 5049–5073 July 9-14, 2023 ©2023 Association for Computational Linguistics NLPEER: A Unified Resource for the Computational Study of Peer Review Nils Dycke, Ilia Kuznetsov, Iryna Gurevych Ubiquitous Knowledge Processing Lab (UKP Lab) Department of Computer Science and Hessian Center for AI (hessian.AI) Technical University of Darmstadt ukp.informatik.tu-darmstadt.de Abstract Peer review constitutes a core component of scholarly publishing; yet it demands substan- tial expertise and training, and is susceptible to errors and biases. Various applications of NLP for peer reviewing assistance aim to support reviewers in this complex process, but the lack of clearly licensed datasets and multi-domain corpora prevent the systematic study of NLP for peer review. To remedy this, we introduce NLPEER – the first ethically sourced multido- main corpus of more than 5k papers and 11k re- view reports from five different venues. In addi- tion to the new datasets of paper drafts, camera- ready versions and peer reviews from the NLP community, we establish a unified data repre- sentation and augment previous peer review datasets to include parsed and structured pa- per representations, rich metadata and version- ing information. We complement our resource with implementations and analysis of three reviewing assistance tasks, including a novel guided skimming task. Our work paves the path towards systematic, multi-faceted, evidence- based study of peer review in NLP and beyond. The data1 and code2 are publicly available. 1 Introduction Research publication is the primary unit of scien- tific communication. To ensure publication quality and to prioritise research outputs, most scientific communities rely on peer review (Johnson et al., 2018) – a distributed procedure where independent referees determine if a manuscript adheres to the standards of the field. Despite its utility and wide application, peer review is an effortful activity that requires expertise and is prone to bias (Tomkins et al., 2017; Lee et al., 2013; Stelmakh et al., 2021). An active line of research in NLP for peer review strive to address these challenges by supporting the 1https://tudatalib.ulb.tu-darmstadt. de/handle/tudatalib/3618 2https://github.com/UKPLab/nlpeer Figure 1: NLPEER unites openly licensed datasets from different research communities, reviewing systems and time periods, including three previously unreleased text collections: ARR-22, COLING-20 and F1000-22. underlying editorial process (e.g. Price and Flach, 2017; Kang et al., 2018; Shah, 2019), decision making (e.g. Shen et al., 2022; Dycke et al., 2021; Ghosal et al., 2019), review writing (Yuan et al., 2022), and by studying review discourse (e.g. Ken- nard et al., 2022; Kuznetsov et al., 2022; Hua et al., 2019; Cheng et al., 2020; Ghosal et al., 2022b). Despite the methodological advancements, sev- eral factors prevent NLP research for peer review at large. The computational study of peer review lacks a (1) solid data foundation: reviewing data is rarely public and comes with legal and ethical challenges; existing sources of peer reviewing data and the derivative datasets are not licensed, which legally prevents reuse and redistribution (Dycke et al., 2022). Peer reviewing practices vary …

Content truncated to 500 words