Aggregating and Predicting Sequence Labels from Crowd Annotations

Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 299–309 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1028 Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 299–309 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1028 Aggregating and Predicting Sequence Labels from Crowd Annotations An T. Nguyen1 Byron C. Wallace2 Junyi Jessy Li3 Ani Nenkova3 Matthew Lease 1 1University of Texas at Austin, 2Northeastern University, 3University of Pennsylvania, atn@cs.utexas.edu, byron@ccs.neu.edu, {ljunyi|nenkova}@seas.upenn.edu, ml@utexas.edu Abstract Despite sequences being core to NLP, scant work has considered how to handle noisy sequence labels from multiple anno- tators for the same text. Given such anno- tations, we consider two complementary tasks: (1) aggregating sequential crowd la- bels to infer a best single set of consen- sus annotations; and (2) using crowd an- notations as training data for a model that can predict sequences in unannotated text. For aggregation, we propose a novel Hid- den Markov Model variant. To predict se- quences in unannotated text, we propose a neural approach using Long Short Term Memory. We evaluate a suite of meth- ods across two different applications and text genres: Named-Entity Recognition in news articles and Information Extraction from biomedical abstracts. Results show improvement over strong baselines. Our source code and data are available online1. 1 Introduction Many important problems in Natural Language Processing (NLP) may be viewed as sequence la- beling tasks, such as part-of-speech (PoS) tagging, named-entity recognition (NER), and Information Extraction (IE). As with other machine learning tasks, automatic sequence labeling typically re- quires annotated corpora on which to train pre- dictive models. While such annotation was tra- ditionally performed by domain experts, crowd- sourcing has become a popular means to acquire large labeled datasets at lower cost, though anno- tations from laypeople may be lower quality than those from domain experts (Snow et al., 2008). It 1 Soure code and biomedical abstract data: www.github.com/thanhan/seqcrowd-acl17, www.byronwallace.com/EBM_abstracts_data is therefore essential to model crowdsourced la- bel quality, both to estimate individual annotator reliability and to aggregate individual annotations to induce a single set of “reference standard” con- sensus labels. While many models have been pro- posed for aggregating crowd labels for binary or multiclass classification problems (Sheshadri and Lease, 2013), far less work has explored crowd- based annotation of sequences (Finin et al., 2010; Hovy et al., 2014; Rodrigues et al., 2014). In this paper, we investigate two complemen- tary challenges in using sequential crowd labels: how to best aggregate them (Task 1); and how to accurately predict sequences in unannotated text given training data from the crowd (Task 2). For aggregation, one might want to induce a single set of high-quality consensus annotations for various purposes: (i) for direct use at run-time (when a given application requires human-level accuracy in identifying sequences); (ii) for sharing with oth- ers; or (iii) for training a predictive model. When human-level accuracy in tagging of se- quences is not …

Content truncated to 500 words