A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications

Proceedings of NAACL-HLT 2018, pages 1647–1661 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications Dongyeop Kang1 Waleed Ammar2 Bhavana Dalvi Mishra2 Madeleine van Zuylen2 Sebastian Kohlmeier2 Eduard Hovy1 Roy Schwartz2,3 1School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA 2Allen Institute for Artificial Intelligence, Seattle, WA, USA 3Paul G. Allen Computer Science & Engineering, University of Washington, Seattle, WA, USA {dongyeok,hovy}@cs.cmu.edu {waleeda,bhavanad,madeleinev,sebastiank,roys}@allenai.org Abstract Peer reviewing is a central component in the scientific publishing process. We present the first public dataset of scien- tific peer reviews available for research pur- poses (PeerRead v1),1 providing an oppor- tunity to study this important artifact. The dataset consists of 14.7K paper drafts and the corresponding accept/reject decisions in top-tier venues including ACL, NIPS and ICLR. The dataset also includes 10.7K textual peer reviews written by experts for a subset of the papers. We describe the data collection process and report interesting ob- served phenomena in the peer reviews. We also propose two novel NLP tasks based on this dataset and provide simple base- line models. In the first task, we show that simple models can predict whether a paper is accepted with up to 21% error reduc- tion compared to the majority baseline. In the second task, we predict the numerical scores of review aspects and show that sim- ple models can outperform the mean base- line for aspects with high variance such as ‘originality’ and ‘impact’. 1 Introduction Prestigious scientific venues use peer reviewing to decide which papers to include in their journals or proceedings. While this process seems essential to scientific publication, it is often a subject of de- bate. Recognizing the important consequences of peer reviewing, several researchers studied various aspects of the process, including consistency, bias, author response and general review quality (e.g., Greaves et al., 2006; Ragone et al., 2011; De Silva and Vance, 2017). For example, the organizers of 1https://github.com/allenai/PeerRead the NIPS 2014 conference assigned 10% of confer- ence submissions to two different sets of reviewers to measure the consistency of the peer reviewing process, and observed that the two committees dis- agreed on the accept/reject decision for more than a quarter of the papers (Langford and Guzdial, 2015). Despite these efforts, quantitative studies of peer reviews had been limited, for the most part, to the few individuals who had access to peer reviews of a given venue (e.g., journal editors and program chairs). The goal of this paper is to lower the barrier to studying peer reviews for the scientific commu- nity by introducing the first public dataset of peer reviews for research purposes: PeerRead. We use three strategies to construct the dataset: (i) We collaborate with conference chairs and con- ference management systems to allow authors and reviewers to opt-in their paper drafts and peer re- views, respectively. (ii) We crawl publicly available peer reviews and annotate textual reviews with nu- merical scores for aspects such as ‘clarity’ and …

Content truncated to 500 words