What Factors Should Paper-Reviewer Assignments Rely On? Community Perspectives on Issues and Ideals in Conference Peer-Review

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4810 - 4823 July 10-15, 2022 ©2022 Association for Computational Linguistics What Factors Should Paper-Reviewer Assignments Rely On? Community Perspectives on Issues and Ideals in Conference Peer-Review Terne Sasha Thorn Jakobsen1, Anna Rogers1,2 1 Copenhagen Center for Social Data Science, University of Copenhagen 2 RIKEN Center for Computational Science 1terne.thorn@sodas.ku.dk, 2arogers@sodas.ku.dk Abstract Both scientific progress and individual re- searcher careers depend on the quality of peer review, which in turn depends on paper- reviewer matching. Surprisingly, this problem has been mostly approached as an automated recommendation problem rather than as a mat- ter where different stakeholders (area chairs, reviewers, authors) have accumulated experi- ence worth taking into account. We present the results of the first survey of the NLP commu- nity, identifying common issues and perspec- tives on what factors should be considered by paper-reviewer matching systems. This study contributes actionable recommendations for im- proving future NLP conferences, and desider- ata for interpretable peer review assignments. 1 Introduction Peer review is increasingly coming under criticism for its arbitrariness. Two NeurIPS experiments (Price, 2014; Cortes and Lawrence, 2021; Beygelz- imer et al., 2021) have shown that the reviewers are good at identifying papers that are clearly bad, but the agreement on the “good” papers appears to be close to random. Among the likely reasons for that are cognitive and social biases of NLP reviewers (see overview by Rogers and Augenstein, 2020), fundamental disagreements in such an interdisci- plinary field as NLP, and acceptance rates that are kept low1 irrespective of the ratio of high-quality submissions. Such arbitrariness leads to understandable frus- tration on the part of the authors whose jobs and graduation depend on publications, and it also means lost time and opportunities (Aczel et al., 2021; Gordon and Poulin, 2009) for science over- all. Reviews written by someone who does not have the requisite expertise, or does not even consider the given type of research as a contribution, it is a 1https://twitter.com/tomgoldsteincs/ status/1388156022112624644 14 (8%) 43 (24%) 16 (9%) 47 (26%) 60 (33%) AC / Action Editor Author Reviewer Figure 1: Overview of all respondents and overlap of their roles for their last experience at NLP venues. loss for all parties: the authors do not get the intel- lectual exchange that could improve their projects and ideas, and reviewers simply lose valuable time without learning something they could use. It is also a loss for the field overall: less popular topics could be systematically disadvantaged, leading to ossification of the field (Chu and Evans, 2021). This paper contributes a snapshot of this prob- lem in NLP venues, based on a survey of authors, reviewers and area chairs (ACs). We collected 180 responses, which is is comparable to the volume of feedback collected for implementing the ACL Rolling Review (ARR). The overall distribution of respondents’ roles is shown in fig. 1. We present the commonly reported issues and community pref- erences …

Content truncated to 500 words