The lack of theory is painful: Modeling Harshness in Peer Review Comments

Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 925–935 November 20–23, 2022. ©2022 Association for Computational Linguistics 925 The lack of theory is painful: Modeling Harshness in Peer Review Comments Rajeev Verma1, Rajarshi Roychowdhury2, Tirthankar Ghosal3 1Independent Researcher 2Jadavpur University, India 3Charles University, Faculty of Mathematics and Physics, Institute of Formal and Applied Linguistics, Czech Republic 1rajeev.ee15@gmail.com, 2rroychoudhury2@gmail.com, 3ghosal@ufal.mff.cuni.cz Abstract The peer-review system has primarily remained the central process of all science communica- tions. However, research has shown that the process manifests a power-imbalance scenario where the reviewer enjoys a position where their comments can be overly critical and wilfully obtuse without being held accountable. This brings into question the sanctity of the peer-review process, turning it into a fraught and traumatic experience for authors. A little more effort to still remain critical but be constructive in the feedback would help foster a progressive outcome from the peer-review process. In this paper, we argue to intervene at the step where this power imbalance actually begins in the system. To this end, we develop the first dataset of peer-review comments with their real-valued harshness scores. popularWe build our dataset by using the Best-Worst-Scaling mechanism. We show the utility of our dataset for text moderation in peer reviews to make review reports less hurtful and more welcoming. We release our dataset and associated codes in https://github.com/Tirthankar-Ghosal/ moderating-peer-review-harshness. Our research is one step towards helping create constructive peer-review reports. 1 Introduction The peer-review system has largely remained the central and universal quality control system in all scientific fields. Hyland and Jiang (2020) argues that the peer-review system embodies Universal- ism and Organized skepticism where the former means “an adherence to objectivity rather than self- interest,” and the latter calls to the spirit that “no theory is accepted merely on the authority of the proponent.” Both these goals are crucial to the success of this science scrutiny system that has been the de-facto method for scientific validation for ages. Nonetheless, the past few years have put this system to stress test with ever-increasing re- search submissions (Ghosal et al., 2019a), a dearth of experienced reviewers, and criticisms like exclu- sionary, arbitrary, inconsistent, etc. being leveled at this fundamental process of science evaluation (Ghosal, 2022). These challenges have the poten- tial to turn this central process into a fraught, and traumatic experience, especially for young authors when the reviewers are overly critical or wilfully obtuse (Wilcox, 2019). In an ever-increasing com- petition in the academic job market, where the ca- reer of researchers depends on the impact and pres- tige of where their work is published, this leads to a natural disdain among the authors for the peer- review process, which is laden with these critical issues. While the peer-review process is by def- inition a process to evaluate the research under submission — a litmus test to separate the sweet from the …

Content truncated to 500 words