Bringing Active Learning to Life

Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), pages 949–957, Beijing, August 2010 Bringing Active Learning to Life Ines Rehbein Computational Linguistics Saarland University Josef Ruppenhofer Computational Linguistics Saarland University {rehbein|josefr|apalmer}@coli.uni-sb.de Alexis Palmer Computational Linguistics Saarland University Abstract Active learning has been applied to dif- ferent NLP tasks, with the aim of limit- ing the amount of time and cost for human annotation. Most studies on active learn- ing have only simulated the annotation scenario, using prelabelled gold standard data. We present the first active learning experiment for Word Sense Disambigua- tion with human annotators in a realistic environment, using fine-grained sense dis- tinctions, and investigate whether AL can reduce annotation cost and boost classifier performance when applied to a real-world task. 1 Introduction Active learning has recently attracted attention as having the potential to overcome the knowledge acquisition bottleneck by limiting the amount of human annotation needed to create training data for statistical classifiers. Active learning has been shown, for a number of different NLP tasks, to re- duce the number of manually annotated instances needed for obtaining a consistent classifier perfor- mance (Hwa, 2004; Chen et al., 2006; Tomanek et al., 2007; Reichart et al., 2008). The majority of such results have been achieved by simulating the annotation scenario using prela- belled gold standard annotations as a stand-in for real-time human annotation. Simulating annota- tion allows one to test different parameter set- tings without incurring the cost of human anno- tation. There is, however, a major drawback: we do not know whether the results of experiments performed using hand-corrected data carry over to real-world scenarios in which individual human annotators produce noisy annotations. In addi- tion, we do not know to what extent error-prone annotations mislead the learning process. A sys- tematic study of the impact of erroneous annota- tion on classifier performance in an active learn- ing (AL) setting is overdue. We need to know a) whether the AL approach can really improve clas- sifier performance and save annotation time when applied in a real-world scenario with noisy data, and b) whether AL works for classification tasks with fine-grained or complex annotation schemes and a low inter-annotator agreement. In this paper we bring active learning to life in the context of frame semantic annotation of Ger- man texts within the SALSA project (Burchardt et al., 2006). Specifically, we apply AL methods for learning to assign semantic frames to predi- cates, following Erk (2005) in treating frame as- signment as a Word Sense Disambiguation task. Under our fine-grained annotation scheme, anno- tators have to deal with a high level of ambigu- ity, resulting in low inter-annotator agreement for some word senses. This fact, along with the po- tential for wrong annotation decisions or possi- ble biases from individual annotators, results in an annotation environment in which we get noisy data which might mislead the classifier. A sec- ond characteristic of our scenario is that there is no gold standard for the newly annotated data, which …

Content truncated to 500 words