Reproducibility in Computational Linguistics: Is Source Code Enough?

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2350 - 2361 December 7-11, 2022 ©2022 Association for Computational Linguistics Reproducibility in Computational Linguistics: Is Source Code Enough? Mohammad Arvan, Luís Pina, and Natalie Parde Department of Computer Science University of Illinois Chicago {marvan3,luispina,parde}@uic.edu Abstract The availability of source code has been put forward as one of the most critical factors for improving the reproducibility of scientific re- search. This work studies trends in source code availability at major computational linguistics conferences; namely, ACL, EMNLP, LREC, NAACL, and COLING. We observe positive trends, especially in conferences that actively promote reproducibility. We follow this by con- ducting a reproducibility study of eight papers published in EMNLP 2021, finding that source code releases leave much to be desired. Mov- ing forward, we suggest all conferences require self-contained artifacts and provide a venue to evaluate such artifacts at the time of publication. Authors can include small-scale experiments and explicit scripts to generate each result to improve the reproducibility of their work. 1 Introduction Modern natural language processing and compu- tational linguistics research progresses at a rapid pace, making it expedient to build upon the ground- work of earlier research rather than reinventing solutions from scratch. This is complicated by the field’s reliance in recent years on deep learn- ing models that are difficult to interpret and sensi- tive to small changes in architecture and environ- ment. These distinct characteristics hinder their reproducibility, as do the substantial computing resources often required to replicate them. Thus, many strong NLP models fall short on two crucial parameters: accessibility and reproducibility. Although deep learning models today allow for effective processing in highly complex search spaces and in most cases outperform solutions from the past, researchers must consider the risks and potential ethical implications associated with their use alongside the performance benefits. There have been numerous calls and attempts (Dodge et al., 2019; Rogers et al., 2021) to push the community into taking more responsibility for the reproducibil- ity of their research. Several venues have intro- duced and adapted reproducibility checklists and standards (Pineau, 2019; Stojnic, 2022) into their submission process (Nature, 2022; NeurIPS, 2022; AAAI, 2022; ACM, 2022; ACL, 2022a; Deutsch et al., 2022), and others have organized repro- ducibility challenges (Sinha et al., 2021; Belz et al., 2021) to encourage the community to reproduce published research. Although this is undeniably a step forward, it is unclear how large of a step it has been. There is also some concern that ask- ing reviewers to evaluate reproducibility burdens them with another time-consuming task that might extend beyond their expertise. In this work, we analyze and report on the state of reproducibility in NLP. We investigate the ex- tent to which content necessary for reproducing research is currently available, and study the influ- ence of reproducibility checklists on this availabil- ity over the last seven years. We also conduct an eight-paper reproducibility case study to develop a deeper understanding of current strengths and weaknesses …

Content truncated to 500 words