Preregistering NLP research

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 613–623 June 6–11, 2021. ©2021 Association for Computational Linguistics 613 Preregistering NLP research Emiel van Miltenburg and Chris van der Lee and Emiel Krahmer Tilburg Center for Cognition and Communnication (TiCC) Tilburg University Tilburg, The Netherlands {C.W.J.vanMiltenburg,C.vdrLee,E.J.Krahmer}@tilburguniversity.edu Abstract Preregistration refers to the practice of specify- ing what you are going to do, and what you ex- pect to find in your study, before carrying out the study. This practice is increasingly com- mon in medicine and psychology, but is rarely discussed in NLP. This paper discusses pre- registration in more detail, explores how NLP researchers could preregister their work, and presents several preregistration questions for different kinds of studies. Finally, we argue in favour of registered reports, which could pro- vide firmer grounds for slow science in NLP research. The goal of this paper is to elicit a discussion in the NLP community, which we hope to synthesise into a general NLP prereg- istration form in future research. 1 Introduction Scientific results are only as reliable as the methods that we use to obtain those results. Recent years have seen growing concerns about the reproducibil- ity of scientific research, leading some to speak of a ‘reproducibility crisis’ (see Fidler and Wilcox 2018 for an overview of the debate). Although the main focus of the debate has been on psychology (e.g. through Open Science Collaboration 2015) and medicine (Macleod et al., 2014), there are wor- ries about the reproducibility of Natural Language Processing (NLP) research as well (Fokkens et al., 2013; Cohen et al., 2018; Moore and Rayson, 2018; Branco et al., 2020). The reproducibility debate has led to Munafò et al.’s (2017) Manifesto for re- producible science, where the authors discuss the different threats to reproducible science, and dif- ferent ways to address these threats. We will first highlight some of their proposals, and discuss their adoption rate in NLP. Our main observation is that preregistration is rarely used. We believe this is an undesirable situation, and devote the rest of this paper to argue for preregistration of NLP research. Munafò et al. recommend more methodolog- ical training, so that e.g. statistical methods are applied correctly. In NLP, we see different re- searchers picking up the gauntlet to teach others about statistics (Dror et al., 2018, 2020), achiev- ing language-independence (Bender, 2011), or best practices in human evaluation (van der Lee et al., 2019, 2021). Moreover, every *ACL conference offers tutorials on a wide range of different top- ics. While efforts to improve methodology could be more systematic (e.g. by actively encouraging methodology tutorials, and working towards com- munity standards),1 the infrastructure is in place. Munafò et al. also recommend to diversify peer review. Instead of only having journals, that are re- sponsible for both the evaluation and dissemination of research, we can now also solicit peer feedback after publishing our work on a platform like ArXiv or OpenReview. The …

Content truncated to 500 words