What Can We Do to Improve Peer Review in NLP?

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1256–1262 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 1256 What Can We Do to Improve Peer Review in NLP? Anna Rogers Center for Social Data Science University of Copenhagen arogers@sodas.ku.dk Isabelle Augenstein Department of Computer Science University of Copenhagen augenstein@di.ku.dk Abstract Peer review is our best tool for judging the quality of conference submissions, but it is be- coming increasingly spurious. We argue that a part of the problem is that the reviewers and area chairs face a poorly defined task forcing apples-to-oranges comparisons. There are sev- eral potential ways forward, but the key dif- ficulty is creating the incentives and mecha- nisms for their consistent implementation in the NLP community. 1 Introduction Traditionally, peer review is expected to act as a filter for high-quality, impactful work (Wingfield, 2018), but this does not hold in practice: • Peer review does not guarantee quality con- trol, neither for small errors nor for serious methodological flaws – even in biomedical lit- erature, where publishing flawed results does real damage (Smith, 2010). • Peer review fails to detect impactful papers. The correlation between conference rejection rates and conference impact in terms of cita- tions is not strong (Freyne et al., 2010; Ragone et al., 2011, 2013), and rejects from one con- ference sometimes receive awards1 at another. The problem is that both expectations are unreal- istic to begin with. A peer reviewer cannot perform real quality control, because that would mean en- suring that a paper is reproducible. Not only is that impossible, only having a few hours to review a pa- per, but it is a general problem for Deep Learning (DL)-based NLP (Crane, 2018; Rogers, 2019). The reproducibility checklist at EMNLP 2020 (Dodge et al., 2019) is the first step in that direction. 1Mani (2011) discusses the example of a paper by Brana- van et al. that received the award at ACL 2009 after being rejected from NAACL (scored at 2.3/5). More recently, ELMo (Peters et al., 2018) received low scores from ICLR reviewers and was resubmitted to NAACL to win the award there. submissions merit acceptance threshold Realistic paper merit distribution, adapted from Anderson (2009) Paper merit distribution, with which peer review could be reliable Figure 1: Paper merit distribution As for paper impact, it is distinct from its scien- tific merit (Bhattacharya and Packalen, 2020), and strongly depends on completely orthogonal factors: how niche is the topic, how much promotion was done, whether the paper offers room to innovate with a low entry barrier2 (Anderson, 2009). What we could realistically expect from peer re- view is to reject the papers with obvious methodol- ogy flaws, and turn the spotlight on the ideas which would be beneficial for the field to discuss. How- ever, the current process is not set up to achieve either purpose. Instead, it aims to rank all submis- sions by their merit so as to identify the top 25%. That task, we argue, …

Content truncated to 500 words