Ensemble-based Active Learning for Parse Selection

Ensemble-based Active Learning for Parse Selection Miles Osborne and Jason Baldridge School of Informatics University of Edinburgh Edinburgh EH8 9LW, UK � miles,jbaldrid � @inf.ed.ac.uk Abstract Supervised estimation methods are widely seen as being superior to semi and fully unsuper- vised methods. However, supervised methods crucially rely upon training sets that need to be manually annotated. This can be very ex- pensive, especially when skilled annotators are required. Active learning (AL) promises to help reduce this annotation cost. Within the complex domain of HPSG parse selection, we show that ideas from ensemble learning can help further reduce the cost of annotation. Our main results show that at times, an ensemble model trained with randomly sampled exam- ples can outperform a single model trained us- ing AL. However, converting the single-model AL method into an ensemble-based AL method shows that even this much stronger baseline model can be improved upon. Our best results show a ��� reduction in annotation cost com- pared with single-model random sampling. 1 Introduction Active learning (AL) methods, such as uncertainty sam- pling (Cohn et al., 1995) or query by committee (Seung et al., 1992), can dramatically reduce the cost of creat- ing an annotated dataset. In particular, they enable rapid creation of labeled datasets which can then be used for trainable speech and language technologies. Progress in AL will therefore translate into even greater savings in annotation costs and hence faster creation of speech and language systems. In this paper, we: � Present a novel way of improving uncertainty sam- pling by generalizing it from using a single model to using an ensemble model. This generalization easily outperforms single-model uncertainty sampling. � Introduce a new, extremely simple AL method (called lowest best probability selection) which is competitive with uncertainty sampling and can also be improved using ensemble techniques. � Show that an ensemble of models trained using ran- domly sampled examples can outperform a single model trained using (single model) AL methods. � Demonstrate further reductions in annotation cost when we train the ensemble parse selection model using examples selected by an ensemble-based ac- tive learner. This result shows that ensemble learn- ing can improve both the underlying model and also the way we select examples for it. Our domain is parse selection for Head-Driven Phrase Structure Grammar (HPSG). Although annotated corpora exist for HPSG, such corpora do not exist in significant volumes and are limited to a few small domains (Oepen et al., 2002). Even if it were possible to bootstrap from the Penn Treebank, it is still unlikely that there would be sufficient quantities of high quality material necessary to improve parse selection for detailed linguistic formalisms such as HPSG. There is thus a pressing need to efficiently create significant volumes of annotated material. AL applied to parse selection is much more challeng- ing than applying it to simpler tasks such as text classifi- cation or part-of-speech tagging. Our labels are complex objects rather than discrete values drawn from a small, fixed set. Furthermore, …

Content truncated to 500 words