A Novel Workflow for Accurately and Efficiently Crowdsourcing Predicate Senses and Argument Labels

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 415–421 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 415 A Novel Workflow for Accurately and Efficiently Crowdsourcing Predicate Senses and Argument Labels Youxuan Jiang1, Huaiyu Zhu2, Jonathan K. Kummerfeld1, Yunyao Li2, Walter Lasecki1 University of Michigan, Ann Arbor1 IBM Research, Almaden2 lyjiang@umich.edu huaiyu@us.ibm.com jkummerf@umich.edu yunyaoli@us.ibm.com wlasecki@umich.edu Abstract Resources for Semantic Role Labeling (SRL) are typically annotated by experts at great ex- pense. Prior attempts to develop crowdsourc- ing methods have either had low accuracy or required substantial expert annotation. We pro- pose a new multi-stage crowd workflow that substantially reduces expert involvement with- out sacrificing accuracy. In particular, we in- troduce a unique filter stage based on the key observation that crowd workers are able to al- most perfectly filter out incorrect options for labels. Our three-stage workflow produces an- notations with 95% accuracy for predicate la- bels and 93% for argument labels, which is comparable to expert agreement. Compared to prior work on crowdsourcing for SRL, we de- crease expert effort by 4x, from 56% to 14% of cases. Our approach enables more scalable an- notation of SRL, and could enable annotation of NLP tasks that have previously been consid- ered too complex to effectively crowdsource. 1 Introduction High quality data is crucial in NLP, but difficult to collect for complex tasks such as semantic role labeling (SRL). Annotating Propbank involved a team of annotators, each of whom took around three days to learn the annotation process (Palmer et al., 2005). For tasks such as sentiment anal- ysis (Socher et al., 2013) and question answer- ing (Rajpurkar et al., 2016), crowdsourcing has produced massive datasets that enabled the devel- opment of new, more sophisticated models. Re- cent work introduced a hybrid workflow to allow crowd workers to usefully contribute to annotation of SRL (Wang et al., 2017), but still required expert annotation in a third of cases. This paper introduces a new hybrid SRL annota- tion workflow with the goal of minimizing expert annotation without sacrificing annotation accuracy. In order to develop our method, we first explored why SRL annotations are hard for crowd workers. We found that workers had difficulty identifying the correct answer because the number of options for labels in SRL can be overwhelming and work- ers lack the linguistic expertise to handle subtle cases. However, we also observed that (1) non- expert workers are capable of reliably identifying many of the answers that are incorrect, and (2) when given the opportunity, crowd workers can accurately identify the limits of their knowledge. Based on these observations, we developed a three phase workflow: (1) workers filter the set of options, reducing the complexity of the task, (2) workers select an answer or say they are unsure, and (3) difficult cases that workers disagreed on or were unsure of are decided by experts. The experts choose from the complete, unfiltered set of options. To measure the effectiveness of the approach we ran experiments at two …

Content truncated to 500 words