Beyond Fair Pay: Ethical Implications of NLP Crowdsourcing

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3758–3769 June 6–11, 2021. ©2021 Association for Computational Linguistics 3758 Beyond Fair Pay: Ethical Implications of NLP Crowdsourcing Boaz Shmueli1,2,3,∗, Jan Fell2, Soumya Ray2, and Lun-Wei Ku3 1Social Networks and Human-Centered Computing, TIGP, Academia Sinica 2Institute of Service Science, National Tsing Hua University 3Institute of Information Science, Academia Sinica Abstract The use of crowdworkers in NLP research is growing rapidly, in tandem with the expo- nential increase in research production in ma- chine learning and AI. Ethical discussion re- garding the use of crowdworkers within the NLP research community is typically confined in scope to issues related to labor conditions such as fair pay. We draw attention to the lack of ethical considerations related to the var- ious tasks performed by workers, including la- beling, evaluation, and production. We find that the Final Rule, the common ethical frame- work used by researchers, did not anticipate the use of online crowdsourcing platforms for data collection, resulting in gaps between the spirit and practice of human-subjects ethics in NLP research. We enumerate common sce- narios where crowdworkers performing NLP tasks are at risk of harm. We thus recom- mend that researchers evaluate these risks by considering the three ethical principles set up by the Belmont Report. We also clarify some common misconceptions regarding the Institu- tional Review Board (IRB) application. We hope this paper will serve to reopen the discus- sion within our community regarding the ethi- cal use of crowdworkers. 1 Revisiting the Ethics of Crowdsourcing The information age brought with it the internet, big data, smartphones, AI, and along with these, a plethora of complex ethical challenges. As a result, there is growing concern and discussion on ethics within the research community at large, including the NLP community. This is manifested in new ethics-focused workshops, ethics conference panels and relevant updates to peer review forms. While ethics in NLP has multiple aspects, most recent attention focuses on pressing issues related to the societal impact of NLP. These include dis- crimination, exclusion, over-generalization, bias, ∗Corresponding author: shmueli@iis.sinica.edu.tw and fairness (Hovy and Spruit, 2016; Leidner and Plachouras, 2017). Other works are concerned with the ethical implications of NLP shared tasks (Parra Escartín et al., 2017), and introducing ethics into the NLP curriculum (Bender et al., 2020). A substantial amount of NLP research now takes advantage of crowdworkers — workers on crowd- sourcing platforms such as Amazon Mechanical Turk (known also as AMT or MTurk), Figure Eight1, Appen, Upwork, Prolific, Hybrid, Tencent Questionnaire, and Baidu Zhongbao, as well as in- ternal crowdsourcing platforms in companies such as Microsoft and Apple. Workers are recruited to la- bel, evaluate, and produce data. In the pre-internet era, such tasks (e.g. part-of-speech (POS) tagging) were done by hiring expert annotators or linguistics students. However, these are now mostly replaced by crowdworkers due to lower costs, convenience, speed, and scalability. Overall, the general consensus in the literature is …

Content truncated to 500 words