Cheap and Fast – But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks

Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 254–263, Honolulu, October 2008. c⃝2008 Association for Computational Linguistics Cheap and Fast — But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks Rion Snow† Brendan O’Connor‡ Daniel Jurafsky§ Andrew Y. Ng† †Computer Science Dept. Stanford University Stanford, CA 94305 {rion,ang}@cs.stanford.edu ‡Dolores Labs, Inc. 832 Capp St. San Francisco, CA 94110 brendano@doloreslabs.com §Linguistics Dept. Stanford University Stanford, CA 94305 jurafsky@stanford.edu Abstract Human linguistic annotation is crucial for many natural language processing tasks but can be expensive and time-consuming. We ex- plore the use of Amazon’s Mechanical Turk system, a significantly cheaper and faster method for collecting annotations from a broad base of paid non-expert contributors over the Web. We investigate five tasks: af- fect recognition, word similarity, recognizing textual entailment, event temporal ordering, and word sense disambiguation. For all five, we show high agreement between Mechani- cal Turk non-expert annotations and existing gold standard labels provided by expert label- ers. For the task of affect recognition, we also show that using non-expert labels for training machine learning algorithms can be as effec- tive as using gold standard annotations from experts. We propose a technique for bias correction that significantly improves annota- tion quality on two tasks. We conclude that many large labeling tasks can be effectively designed and carried out in this method at a fraction of the usual expense. 1 Introduction Large scale annotation projects such as TreeBank (Marcus et al., 1993), PropBank (Palmer et al., 2005), TimeBank (Pustejovsky et al., 2003), FrameNet (Baker et al., 1998), SemCor (Miller et al., 1993), and others play an important role in natural language processing research, encouraging the development of novel ideas, tasks, and algo- rithms. The construction of these datasets, how- ever, is extremely expensive in both annotator-hours and financial cost. Since the performance of many natural language processing tasks is limited by the amount and quality of data available to them (Banko and Brill, 2001), one promising alternative for some tasks is the collection of non-expert annotations. In this work we explore the use of Amazon Me- chanical Turk1 (AMT) to determine whether non- expert labelers can provide reliable natural language annotations. We chose five natural language under- standing tasks that we felt would be sufficiently nat- ural and learnable for non-experts, and for which we had gold standard labels from expert labelers, as well as (in some cases) expert labeler agree- ment information. The tasks are: affect recogni- tion, word similarity, recognizing textual entailment, event temporal ordering, and word sense disam- biguation. For each task, we used AMT to annotate data and measured the quality of the annotations by comparing them with the gold standard (expert) la- bels on the same data. Further, we compare machine learning classifiers trained on expert annotations vs. non-expert annotations. In the next sections of the paper we introduce the five tasks and the evaluation metrics, and offer methodological insights, including a technique for bias correction that improves annotation quality.2 …

Content truncated to 500 words