Lessons Learned from a Citizen Science Project for Natural Language Processing

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 3594–3608 May 2-6, 2023 ©2023 Association for Computational Linguistics Lessons Learned from a Citizen Science Project for Natural Language Processing Jan-Christoph Klie1 Ji-Ung Lee1 Kevin Stowe1,2 Gözde Gül ¸Sahin1,3 Nafise Sadat Moosavi1,4 Luke Bates1 Dominic Petrak1 Richard Eckart de Castilho1 Iryna Gurevych1 1Ubiquitous Knowledge Processing Lab (UKP Lab) Department of Computer Science and Hessian Center for AI (hessian.AI) Technical University of Darmstadt www.ukp.tu-darmstadt.de 2Educational Testing Service 3KUIS AI, Koç University 4Department of Computer Science, The University of Sheffield Abstract Many Natural Language Processing (NLP) sys- tems use annotated corpora for training and evaluation. However, labeled data is often costly to obtain and scaling annotation projects is difficult, which is why annotation tasks are often outsourced to paid crowdworkers. Citi- zen Science is an alternative to crowdsourcing that is relatively unexplored in the context of NLP. To investigate whether and how well Citi- zen Science can be applied in this setting, we conduct an exploratory study into engaging dif- ferent groups of volunteers in Citizen Science for NLP by re-annotating parts of a pre-existing crowdsourced dataset. Our results show that this can yield high-quality annotations and at- tract motivated volunteers, but also requires considering factors such as scalability, partic- ipation over time, and legal and ethical issues. We summarize lessons learned in the form of guidelines and provide our code and data to aid future work on Citizen Science.1 1 Introduction Data labeling or annotation is often a difficult, time- consuming, and therefore expensive task. Anno- tations are typically drawn from domain experts or are crowdsourced. While experts can produce high-quality annotated data, they are expensive and do not scale well due to their relatively low number (Sorokin and Forsyth, 2008). In contrast, crowdsourcing can be relatively cheap, fast, and scalable, but is potentially less suited for more com- plicated annotation tasks (Drutsa et al., 2020). An- other approach is using Citizen Science, which 1https://github.com/UKPLab/ eacl2023-citizen-science-lessons-learned describes the participation and collaboration of vol- unteers from the general public with researchers to conduct science (Haklay et al., 2021). Over the past decade, Citizen Science platforms, which rely on unpaid volunteers to solve scientific problems, have been used for a wide variety of tasks requir- ing human annotation (Hand, 2010), e.g., classify- ing images of galaxies (Lintott et al., 2008) or for weather observation (Leeper et al., 2015). While Citizen Science has been shown to pro- duce high-quality annotations in ecological or en- vironmental projects (Kosmala et al., 2016), its po- tential has so far not been investigated in depth for Natural Language Processing (NLP). Our goal in this work is to assess the practicality of undertaking annotation campaigns for NLP via Citizen Science. We analyze whether volunteers actually react to our calls and participate, how the resulting quality is compared to crowdsourcing, what the benefits and shortcomings are and what needs to be taken into account when conducting such a project. We especially are interested in …

Content truncated to 500 words