Mining Crowdsourcing Problems from Discussion Forums of Workers

Proceedings of the 28th International Conference on Computational Linguistics, pages 6264–6276 Barcelona, Spain (Online), December 8-13, 2020 6264 Mining Crowdsourcing Problems from Discussion Forums of Workers Zahra Nouri and Henning Wachsmuth and Gregor Engels Department of Computer Science, Paderborn University, Paderborn, Germany znouri@mail.upb.de henningw@upb.de engels@upb.de Abstract Crowdsourcing is used in academia and industry to solve tasks that are easy for humans but hard for computers, in natural language processing mostly to annotate data. The quality of annota- tions is affected by problems in the task design, task operation, and task evaluation that workers face with requesters in crowdsourcing processes. To learn about the major problems, we provide a short but comprehensive survey based on two complementary studies: (1) a literature review where we collect and organize problems known from interviews with workers, and (2) an empir- ical data analysis where we use topic modeling to mine workers’ complaints from a new English corpus of workers’ forum discussions. While literature covers all process phases, problems in the task evaluation are prevalent, including unfair rejections, late payments, and unjustified block- ings of workers. According to the data, however, poor task design in terms of malfunctioning environments, bad workload estimation, and privacy violations seems to bother the workers most. Our findings form the basis for future research on how to improve crowdsourcing processes. 1 Introduction Crowdsourcing refers to the idea of providing collaborative services using the intelligence, skills, and creativity of a huge crowd (Howe, 2006). It has become a prevalent method in academia and industry to solve tasks that are complex for computers and yet easy for humans with low cost and high creativ- ity. A typical use case in natural language processing (NLP) is the annotation of data. Crowdsourcing platforms create an extensive network of people and enable organizations and individuals (the so called task requesters) to hire other individuals (the crowdworkers or just workers) in an anonymous and distant manner. Among the most widely used platforms are Upwork, Appen, and above all Amazon Mechanical Turk (MTurk) which host annotation tasks and collect huge sets of annotated data from workers. Conceptually, the general crowdsourcing process includes three main phases which are illustrated in Figure 1 and further detailed in Section 2: (1) In the task design, requesters create and post tasks on the given platform. (2) In the task operation, workers take on and solve tasks to then submit their results. Workers may ask for clarifications of task details, and requesters may give feedback to ensure correct results (Gupta et al., 2014). And (3) in the task evaluation, requesters decide whether to accept results and hence to pay workers, while workers may ask about feedback, payment, or reasons for rejections. To achieve mutual benefit in this process, workers need to understand the tasks designed by the requesters and their expected results. In practice, however, workers face diverse problems in the process and the interactions (Bederson and Quinn, 2011). This does not only affect the workers’ feelings of pride and sat- isfaction …

Content truncated to 500 words