Learning Whom to Trust with MACE

Proceedings of NAACL-HLT 2013, pages 1120–1130, Atlanta, Georgia, 9–14 June 2013. c⃝2013 Association for Computational Linguistics Learning Whom to Trust with MACE Dirk Hovy1 Taylor Berg-Kirkpatrick2 Ashish Vaswani1 Eduard Hovy3 (1) Information Sciences Institute, University of Southern California, Marina del Rey (2) Computer Science Division, University of California at Berkeley (3) Language Technology Institute, Carnegie Mellon University, Pittsburgh {dirkh,avaswani}@isi.edu, tberg@cs.berkeley.edu, hovy@cmu.edu Abstract Non-expert annotation services like Amazon’s Mechanical Turk (AMT) are cheap and fast ways to evaluate systems and provide categor- ical annotations for training data. Unfortu- nately, some annotators choose bad labels in order to maximize their pay. Manual iden- tification is tedious, so we experiment with an item-response model. It learns in an un- supervised fashion to a) identify which an- notators are trustworthy and b) predict the correct underlying labels. We match perfor- mance of more complex state-of-the-art sys- tems and perform well even under adversarial conditions. We show considerable improve- ments over standard baselines, both for pre- dicted label accuracy and trustworthiness es- timates. The latter can be further improved by introducing a prior on model parameters and using Variational Bayes inference. Ad- ditionally, we can achieve even higher accu- racy by focusing on the instances our model is most confident in (trading in some recall), and by incorporating annotated control instances. Our system, MACE (Multi-Annotator Compe- tence Estimation), is available for download1. 1 Introduction Amazon’s MechanicalTurk (AMT) is frequently used to evaluate experiments and annotate data in NLP (Callison-Burch et al., 2010; Callison-Burch and Dredze, 2010; Jha et al., 2010; Zaidan and Callison-Burch, 2011). However, some turkers try to maximize their pay by supplying quick answers that have nothing to do with the correct label. We refer to 1Available under http://www.isi.edu/ publications/licensed-sw/mace/index.html this type of annotator as a spammer. In order to mit- igate the effect of spammers, researchers typically collect multiple annotations of the same instance so that they can, later, use de-noising methods to infer the best label. The simplest approach is majority voting, which weights all answers equally. Unfor- tunately, it is easy for majority voting to go wrong. A common and simple spammer strategy for cate- gorical labeling tasks is to always choose the same (often the first) label. When multiple spammers follow this strategy, the majority can be incorrect. While this specific scenario might seem simple to correct for (remove annotators that always produce the same label), the situation grows more tricky when spammers do not annotate consistently, but in- stead choose labels at random. A more sophisticated approach than simple majority voting is required. If we knew whom to trust, and when, we could reconstruct the correct labels. Yet, the only way to be sure we know whom to trust is if we knew the correct labels ahead of time. To address this circular problem, we build a generative model of the annotation process that treats the correct labels as latent variables. We then use unsupervised learning to estimate parameters directly from redundant annotations. This is a common approach …

Content truncated to 500 words