The Myth of Double-Blind Review Revisited: ACL vs. EMNLP

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 2317–2327, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 2317 The Myth of Double-Blind Review Revisited: ACL vs. EMNLP Cornelia Caragea1, Ana Sabina Uban2, and Liviu P. Dinu2 1Computer Science, University of Illinois at Chicago Chicago, Illinois 2Faculty of Mathematics and Computer Science, University of Bucharest Bucharest, Romania cornelia@uic.edu, ana.uban@gmail.com, ldinu@fmi.unibuc.ro Abstract The review and selection process for scientific paper publication is essential for the quality of scholarly publications in a scientific field. The double-blind review system, which en- forces author anonymity during the review pe- riod, is widely used by prestigious conferences and journals to ensure the integrity of this pro- cess. Although the notion of anonymity in the double-blind review has been questioned be- fore, the availability of full text paper collec- tions brings new opportunities for exploring the question: Is the double-blind review pro- cess really double-blind? We study this ques- tion on the ACL and EMNLP paper collec- tions and present an analysis on how well deep learning techniques can infer the authors of a paper. Specifically, we explore Convolutional Neural Networks trained on various aspects of a paper, e.g., content, style features, and ref- erences, to understand the extent to which we can infer the authors of a paper and what as- pects contribute the most. Our results show that the authors of a paper can be inferred with accuracy as high as 87% on ACL and 78% on EMNLP for the top 100 most prolific authors. 1 Introduction The scientific peer-review process is indispens- able for the dissemination of high-quality in- formation (Hojat et al., 2003). However, one of the major problems with this process is bias (Williamson, 2003; Tomkins et al., 2017). For ex- ample, Tomkins et al. (2017) performed an exper- iment during the ACM Web Search and Data Min- ing conference 2017 to understand the potential bias in favoring authors from prestigious institu- tions and found that, indeed, when reviewers have access to the authors identities, they more often tend to favor well known authors from prestigious institutions. The double-blind review process is generally employed by top scientific journals and conferences in order to guarantee fairness of the paper selection, and thus, plays an essential role in how scientific quality is eventually measured (Meadows, 1998). It is designed to reduce the risk of bias in paper reviews, ensuring that all papers are judged solely based on their content and intrin- sic quality and that any author has a fair chance of having a paper accepted, regardless of their pres- tige or previous work. The double-blind review process implies that the submitted papers have to be anonymized, i.e., the authors’ names are not ex- plicitly available with the papers, and any direct or indirect indications of who the authors might be (for example, referring to self-citations in the first person) are forbidden. Reviewers have ac- cess …

Content truncated to 500 words