What Ingredients Make for an Effective Crowdsourcing Protocol for Difficult NLU Data Collection Tasks?

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 1221–1235 August 1–6, 2021. ©2021 Association for Computational Linguistics 1221 What Ingredients Make for an Effective Crowdsourcing Protocol for Difficult NLU Data Collection Tasks? Nikita Nangia1∗ Saku Sugawara2∗ Harsh Trivedi3 Alex Warstadt1 Clara Vania4† Samuel R. Bowman1 1New York University, 2National Institute of Informatics, 3Stony Brook University, 4Amazon Correspondence: {nikitanangia, bowman}@nyu.edu, saku@nii.ac.jp Abstract Crowdsourcing is widely used to create data for common natural language understanding tasks. Despite the importance of these datasets for measuring and refining model understand- ing of language, there has been little focus on the crowdsourcing methods used for collecting the datasets. In this paper, we compare the effi- cacy of interventions that have been proposed in prior work as ways of improving data qual- ity. We use multiple-choice question answer- ing as a testbed and run a randomized trial by assigning crowdworkers to write questions un- der one of four different data collection proto- cols. We find that asking workers to write ex- planations for their examples is an ineffective stand-alone strategy for boosting NLU exam- ple difficulty. However, we find that training crowdworkers, and then using an iterative pro- cess of collecting data, sending feedback, and qualifying workers based on expert judgments is an effective means of collecting challenging data. But using crowdsourced, instead of ex- pert judgments, to qualify workers and send feedback does not prove to be effective. We observe that the data from the iterative proto- col with expert assessments is more challeng- ing by several measures. Notably, the human– model gap on the unanimous agreement por- tion of this data is, on average, twice as large as the gap for the baseline protocol data. 1 Introduction Crowdsourcing is a scalable method for construct- ing examples for many natural language processing tasks. Platforms like Amazon’s Mechanical Turk give researchers access to a large, diverse pool of people to employ (Howe, 2006; Snow et al., 2008; Callison-Burch, 2009). Given the ease of data col- lection with crowdsourcing, it has been frequently ∗Equal contribution. †Work done while at New York University. used for collecting datasets for natural language understanding (NLU) tasks like question answer- ing (Mihaylov et al., 2018), reading comprehension (Rajpurkar et al., 2016; Huang et al., 2019), natural language inference (Dagan et al., 2005; Bowman et al., 2015; Williams et al., 2018; Nie et al., 2020a), and commonsense reasoning (Talmor et al., 2019). There has been substantial research devoted to studying crowdsourcing methods, especially in the human-computer interaction literature (Kittur et al., 2008, 2011; Bernstein et al., 2012). However, most prior research investigates methods for collecting accurate annotations for existing data, for example labeling objects in images or labeling the sentiment of sentences (Hsueh et al., 2009; Liu et al., 2019a; Sun et al., 2020). There are some small-scale stud- ies that use writing tasks, like writing product re- views, to compare crowdsourcing methodologies (Dow et al., 2012). However, …

Content truncated to 500 words