DialCrowd 2.0: A Quality-Focused Dialog System Crowdsourcing Toolkit

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 1256–1263 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 1256 DialCrowd 2.0: A Quality-Focused Dialog System Crowdsourcing Toolkit Jessica Huynh, Ting-Rui Chiang, Jeffrey Bigham, Maxine Eskenazi Carnegie Mellon University Pittsburgh, PA {jhuynh, tingruic}@cs.cmu.edu, {jbigham, max}@cmu.edu Abstract Dialog system developers need high-quality data to train, fine-tune and assess their systems. They often use crowdsourcing for this since it provides large quantities of data from many workers. However, the data may not be of sufficiently good quality. This can be due to the way that the requester presents a task and how they interact with the workers. This paper introduces DialCrowd 2.0 to help requesters obtain higher quality data by, for example, presenting tasks more clearly and facilitating effective communication with workers. DialCrowd 2.0 guides developers in creating improved Human Intelligence Tasks (HITs) and is directly applicable to the workflows used currently by developers and researchers. Keywords: dialog, crowdsourcing 1. Introduction High-quality human data is essential in the develop- ment of dialog systems. Many researchers create HITs on crowdsourcing platforms such as Amazon Mechan- ical Turk (AMT) to collect data from humans. Obtain- ing high-quality data is dependent on the usability of the tasks workers are asked to complete (e.g., learnabil- ity, feedback, etc.) (Nielsen, 1994), yet many tasks fall short (Huynh et al., 2021). To address this problem, we introduce DialCrowd 2.0, a substantial update to DialCrowd 1.0. DialCrowd 1.0 (Lee et al., 2018) facilitated data collection by pro- viding an interface that requesters used to create HITs from pre-configured templates. The goal in the 1.0 ver- sion was to make the task creation process more effi- cient. Once a HIT was created, workers accessed and worked on the HIT from the DialCrowd-generated in- terface. The added efficacy that DialCrowd provides was studied with 10 participant/requesters. All par- ticipants observed that DialCrowd shortened the time spent creating the study, and when asked to rate the usefulness of this toolkit, participants responded with an average of 4 on a scale of 1 to 5, with 5 being best. Whereas DialCrowd 1.0 focused on helping requesters create HITs more efficiently, DialCrowd 2.0 addresses factors related to interaction and communication with the workers that can affect the quality of the data ob- tained from a HIT. We have demonstrated the commu- nity’s need for help with these two aspects in a recent study (Huynh et al., 2021). In this study, we looked at the tasks running on AMT over seven consecutive days in August 2021 to analyse their overall quality. The study examined only the natural language processing HITs (excluding computer vision, surveys, etc.) that were presented to workers at this time. In the same way that requesters can give ratings to an individual worker, workers also rate the requesters and share information about them on their crowdsourcing forums and blogs (Paolacci et al., 2010). A high requester rating will attract more good workers while poor …

Content truncated to 500 words