Finding Patterns in Noisy Crowds: Regression-based Annotation Aggregation for Crowdsourced Data

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1907–1912 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Finding Patterns in Noisy Crowds: Regression-based Annotation Aggregation for Crowdsourced Data Natalie Parde and Rodney D. Nielsen Department of Computer Science and Engineering University of North Texas {natalie.parde,rodney.nielsen}@unt.edu Abstract Crowdsourcing offers a convenient means of obtaining labeled data quickly and inex- pensively. However, crowdsourced labels are often noisier than expert-annotated data, making it difficult to aggregate them meaningfully. We present an aggregation approach that learns a regression model from crowdsourced annotations to predict aggregated labels for instances that have no expert adjudications. The predicted la- bels achieve a correlation of 0.594 with expert labels on our data, outperforming the best alternative aggregation method by 11.9%. Our approach also outperforms the alternatives on third-party datasets. 1 Introduction Publicly-available labeled datasets are scarce for many NLP tasks, and crowdsourcing services such as Amazon Mechanical Turk1 (AMT) offer re- searchers a quick, inexpensive means of labeling their data. However, workers employed by these services are typically unfamiliar with the anno- tation tasks, and they may have little motivation to perform high-quality work due to factors such as low pay and anonymity. To further complicate matters, some workers may produce spam or ma- licious responses. Thus, it is not uncommon for workers to correlate poorly with one another. Researchers using crowdsourcing services com- monly aggregate the labels they receive via sim- ple strategies such as using the majority or av- erage label. These methods are best suited for simple, straightforward tasks; with noisier data such as that which may be obtained for more dif- ficult or subjective tasks, these strategies may pro- duce skewed labels that misrepresent the instance. 1www.mturk.com Thus, it is desirable to devise more effective aggre- gation strategies that consider factors such as label distribution and worker quality, while still avoid- ing manual adjudication of all instances. In this work, our contributions are as follows: (1) we develop a regression-based method for au- tomatically aggregating crowdsourced annotations of varying quality, with poor agreement and mini- mal expert-adjudicated data, that addresses multi- ple potential flaws or biases in non-expert human annotation. To do so, we (2) crowdsource anno- tations for a difficult NLP task, metaphor novelty scoring, and (3) describe a process by which we automatically detect untrustworthy workers. We then (4) introduce a feature set that captures label distribution and trustworthiness, and extract the features from our crowdsourced annotations. Fi- nally, (5) we train a regression model that predicts aggregated labels for unseen instances and com- pare the predictions to expert annotations, finding that our method outperforms the best alternative approach. We evaluate our approach both on our data and on existing crowdsourcing datasets. All datasets and source code are available for the re- search community to improve on our results.2 2 Related Work Several methods have been proposed to identify low-quality workers in crowdsourced data. Jaga- bathula et al. (2016) filtered adversarial workers in binary labeling …

Content truncated to 500 words