A Major Obstacle for NLP Research: Let’s Talk about Time Allocation!

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8959 - 8969 December 7-11, 2022 ©2022 Association for Computational Linguistics A Major Obstacle for NLP Research: Let’s Talk about Time Allocation! Katharina Kann♠ and Shiran Dudy♠ and Arya D. McCarthy♣ ♠University of Colorado Boulder firstname.lastname@colorado.edu ♣Johns Hopkins University lastname@jhu.edu Abstract The field of natural language processing (NLP) has grown over the last few years: conferences have become larger, we have published an in- credible amount of papers, and state-of-the-art research has been implemented in a large va- riety of customer-facing products. However, this paper argues that we have been less suc- cessful than we should have been and reflects on where and how the field fails to tap its full potential. Specifically, we demonstrate that, in recent years, subpar time allocation has been a major obstacle for NLP research. We out- line multiple concrete problems together with their negative consequences and, importantly, suggest remedies to improve the status quo. We hope that this paper will be a starting point for discussions around which common practices are – or are not – beneficial for NLP research. 1 Introduction Why did I get nothing done today? is a question many people ask themselves frequently throughout their professional careers. Psychologists agree on good time management skills being of utmost importance for a healthy and productive lifestyle (Lakei, 1973; Claessens et al., 2007; Major et al., 2002, inter alia). However, many academics and industry researchers lack time management skills, working long days and getting not enough done – not even the interesting experiment they had wanted to start over a year ago. In this position paper, we argue that natural lan- guage processing (NLP) as a field has a similar problem: we do not allocate our time well. Instead, we spend it on things that seem more urgent than they are, are easy but unimportant, or result in the largest short-term gains. This paper identifies the largest traps the authors believe the NLP community falls into. We then provide, for each of the four iden- tified problems (P1–P4), suggested remedies. While we know that – just as for individuals – change takes time, we hope that this paper, in combination with 0 1 2 3 4 5 Figure 1: Avg. # of authors per paper; 2000–2021. the EMNLP 2022 special theme Open questions, major obstacles, and unresolved issues in NLP, will ignite critical discussions. Related Work Over the last couple of years, mul- tiple papers have provided critical reflections on the state of affairs in NLP research: Bender and Koller (2020) criticizes the hype around language models and argues, similarly to Bisk et al. (2020), that true understanding is impossible when language is detached from the physical world. In contrast, Bowman (2022) talks about the risks associated with underclaiming. Turning to evaluation, Bowman and Dahl (2021) provides a critical view on benchmark- ing, and Rodriguez et al. (2021) proposes ways to improve leaderboards in order to truly track progress. Other …

Content truncated to 500 words