Active Learning and the Total Cost of Annotation

Active Learning and the Total Cost of Annotation Jason Baldridge and Miles Osborne School of Informatics University of Edinburgh Edinburgh EH8 9LW, UK {jbaldrid,miles}@inf.ed.ac.uk Abstract Active learning (AL) promises to reduce the cost of annotating labeled datasets for trainable human language technologies. Contrary to expectations, when creating labeled training material for HPSG parse selection and later reusing it with other models, gains from AL may be negligible or even negative. This has serious impli- cations for using AL, showing that addi- tional cost-saving strategies may need to be adopted. We explore one such strategy: using a model during annotation to auto- mate some of the decisions. Our best re- sults show an 80% reduction in annotation cost compared with labeling randomly se- lected data with a single model. 1 Introduction AL methods such as uncertainty sampling (Cohn et al., 1995) or query by committee (Seung et al., 1992) have all been shown to dramatically reduce the cost of creating highly informative labeled sets for speech and language technologies. However, ex- periments using AL assume a model that is fixed ahead of time: the model used in AL is the same one we are currently developing training material for. For many complex tasks, we are unlikely to have a clear idea how best to model the task at the time of annotation; thus, in practice, we will need to reuse the labeled training material with other models. In this paper, we show that AL can be brittle: un- der a variety of natural reuse scenarios (for example, allowing the later model to improve in quality, or else reusing the labeled training material using a dif- ferent machine learning algorithm) performance of later models can be significantly undermined when training upon material created using AL. The key to knowing how well one model will be able to use material selected by another is their relatedness – yet there may be no means to determine this prior to an- notation, leading to a chicken-and-egg problem. Our reusability results thus demonstrate that, ad- ditionally, other strategies must be adopted to en- sure we reduce the total cost of annotation. In Os- borne and Baldridge (2004), we showed that ensem- ble models can increase model performance and also produce annotation savings when incorporated into the AL process. An obvious next step is automating some decisions. Here, we consider a simple automa- tion strategy that reduces annotation costs indepen- dently of AL and examine its effect on reusability. We find that using both semi-automation and AL with high-quality models can eliminate the perfor- mance gap found in many reuse scenarios. However, for weak models, we show that semi-automation with random sampling is more effective for improv- ing reusability than using it with AL – demonstrat- ing further cause for caution with AL. Finally, we show that under the standard assump- tion of reuse by the selecting model, using a strat- egy which combines AL, ensembling, and semi- automated annotation, we are able to achieve our highest …

Content truncated to 500 words