NLP Reproducibility For All: Understanding Experiences of Beginners

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 10199–10219 July 9-14, 2023 ©2023 Association for Computational Linguistics NLP Reproducibility For All: Understanding Experiences of Beginners Shane Storks Keunwoo Peter Yu Ziqiao Ma Joyce Chai Computer Science and Engineering Division, University of Michigan {sstorks, kpyu, marstin, chaijy}@umich.edu Abstract As natural language processing (NLP) has re- cently seen an unprecedented level of excite- ment, and more people are eager to enter the field, it is unclear whether current research reproducibility efforts are sufficient for this group of beginners to apply the latest devel- opments. To understand their needs, we con- ducted a study with 93 students in an introduc- tory NLP course, where students reproduced the results of recent NLP papers. Surprisingly, we find that their programming skill and com- prehension of research papers have a limited impact on their effort spent completing the ex- ercise. Instead, we find accessibility efforts by research authors to be the key to success, in- cluding complete documentation, better coding practice, and easier access to data files. Going forward, we recommend that NLP researchers pay close attention to these simple aspects of open-sourcing their work, and use insights from beginners’ feedback to provide actionable ideas on how to better support them. 1 Introduction As natural language processing (NLP) research continues to grab public attention and excitement, it becomes increasingly important for it to be ac- cessible to a broad audience. While the research community works to democratize NLP, it remains unclear whether beginners in the field can easily apply the latest developments. How easy is it for them to reproduce experimental results? Will their programming background or comprehension of pa- pers play a role? What key elements affect their experience here? To address these questions, we conducted a con- trolled user study in an introductory NLP class. We first identified and successfully reproduced the results of three recent NLP publications our- selves. We surveyed students on their background in machine learning and programming, then di- vided them into three skill level groups based on the survey results. Each group was asked to repro- duce the results of the three papers. As students conducted experiments, they tracked their own ef- forts, while their computational resource usage was logged by a shared high-performance computing system. After conducting the experiments, students answered questions and provided feedback about their experiences. Our results show that beginners’ technical skill level and comprehension of the papers play only a small part in their experience in reproducing the results, and we observed a strikingly wide range of time spent on the exercise regardless of these user-specific factors. Meanwhile, we show that re- producibility efforts by paper authors make a much larger impact on user experience, and based on di- rect feedback from these beginners, we find that they encounter a range of roadblocks related to the documentation and ease of use of open-sourced ma- terials. These findings shed light on the extra steps NLP researchers …

Content truncated to 500 words