Use of Formal Ethical Reviews in NLP Literature: Historical Trends and Current Practices

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4704–4710 August 1–6, 2021. ©2021 Association for Computational Linguistics 4704 Use of Formal Ethical Reviews in NLP Literature: Historical Trends and Current Practices Sebastin Santy♠ Anku Rani♥ Monojit Choudhury♠ ♠ Microsoft Research, Bangalore, India ♥ Plaksha University, Mohali, India ♠{t-sesan, monojitc}@microsoft.com, ♥anku.rani@plaksha.org Abstract Ethical aspects of research in language tech- nologies have received much attention recently. It is a standard practice to get a study involving human subjects reviewed and approved by a professional ethics committee/board of the in- stitution. How commonly do we see mention of ethical approvals in NLP research? What types of research or aspects of studies are usu- ally subject to such reviews? With the rising concerns and discourse around the ethics of NLP, do we also observe a rise in formal ethi- cal reviews of NLP studies? And, if so, would this imply that there is a heightened awareness of ethical issues that was previously lacking? We aim to address these questions by conduct- ing a detailed quantitative and qualitative anal- ysis of the ACL Anthology, as well as com- paring the trends in our field to those of other related disciplines, such as cognitive science, machine learning, data mining, and systems. 1 Introduction With the rapid advances in the field of Natural Lan- guage Processing (NLP), language technologies are getting woven into the daily fabric of our lives and the society. Since “language is a portal of emotions, a proxy of human behavior, and a strong signal of individual characteristics” (Hovy and Spruit, 2016), large-scale deployment of language technology has potential risks that require early detection and miti- gation. Naturally, there have been several discus- sions about the potential harms and ethical issues concerning NLP (Hovy and Spruit, 2016; Conway and O’Connor, 2016). They have mostly revolved around building or deploying systems in sensitive areas such as hate speech (Sap et al., 2019), social media (Benton et al., 2017), clinical NLP and men- tal health (ˇSuster et al., 2017; Mikal et al., 2016) and use of sensitive or personal information (Lar- son, 2017). While building NLP systems, there are 2006 2008 2010 2012 2014 2016 2018 2020 Iterations of NLP conferences, workshops and journals 0 1 2 3 4 5 Papers mentioning X terms (%) IRB-related terms *ethic* terms Figure 1: Percentage(%) of papers mentioning IRB-related and *ethic* terms in NLP conferences, journals, and work- shops from 2006 to 2020. also ethical risks associated with involvement of human subjects through user studies or data collec- tion activities (Shmueli et al., 2021). The awareness of the dangers of the existing and new NLP applications has led to the curation of several ethical guidelines and frameworks. Under- girded by lessons from the past, these guidelines and frameworks help researchers consider and con- textualize critical ethical concerns. Most of the ethical issues in NLP are rooted in the data being used for research. Couillault et al. (2014) is one of the first works to explore the ethics …

Content truncated to 500 words