Active Learning with Amazon Mechanical Turk

Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 1546–1556, Edinburgh, Scotland, UK, July 27–31, 2011. c⃝2011 Association for Computational Linguistics Active Learning with Amazon Mechanical Turk Florian Laws Christian Scheible Hinrich Sch¨utze Institute for Natural Language Processing Universit¨at Stuttgart {lawsfn, scheibcn}@ims.uni-stuttgart.de Abstract Supervised classification needs large amounts of annotated training data that is expensive to create. Two approaches that reduce the cost of annotation are active learning and crowd- sourcing. However, these two approaches have not been combined successfully to date. We evaluate the utility of active learning in crowdsourcing on two tasks, named entity recognition and sentiment detection, and show that active learning outperforms random selec- tion of annotation examples in a noisy crowd- sourcing scenario. 1 Introduction Supervised classification is the predominant tech- nique for a large number of natural language pro- cessing (NLP) tasks. The large amount of labeled training data that supervised classification relies on is time-consuming and expensive to create, espe- cially when experts perform the data annotation. Recently, crowdsourcing services like Amazon Me- chanical Turk (MTurk) have become available as an alternative that offers acquisition of non-expert an- notations at low cost. MTurk is a software service that outsources small annotation tasks – called HITs – to a large group of freelance workers. The cost of MTurk annotation is low, but a consequence of us- ing non-expert annotators is much lower annotation quality. This requires strategies for quality control of the annotations. Another promising approach to the data acqui- sition bottleneck for supervised learning is active learning (AL). AL reduces annotation effort by set- ting up an annotation loop where, starting from a small seed set, only the maximally informative ex- amples are chosen for annotation. With these an- notated examples, the classifier is then retrained to again select more informative examples for further annotation. In general, AL needs a lot fewer anno- tations to achieve a desired performance level than random sampling. AL has been successfully applied to a number of NLP tasks such as part-of-speech tagging (Ringger et al., 2007), parsing (Osborne and Baldridge, 2004), text classification (Tong and Koller, 2002), senti- ment detection (Brew et al., 2010), and named entity recognition (NER) (Tomanek et al., 2007). Until recently, most AL studies focused on simulating the annotation process by using already available gold standard data. In reality, however, human annota- tors make mistakes, leading to noise in the annota- tions. For this reason, some authors have questioned the applicability of AL to noisy annotation scenarios such as MTurk (Baldridge and Palmer, 2009; Re- hbein et al., 2010). AL and crowdsourcing are complementary ap- proaches: AL reduces the number of annotations used while crowdsourcing reduces the cost per an- notation. Combined, the two approaches could sub- stantially lower the cost of creating training sets. Our main contribution in this paper is that we show for the first time that AL is significantly bet- ter than randomly selected annotation examples in a real crowdsourcing annotation scenario. Our experiments directly …

Content truncated to 500 words