An Intrinsic Stopping Criterion for Committee-Based Active Learning

Proceedings of the Thirteenth Conference on Computational Natural Language Learning (CoNLL), pages 138–146, Boulder, Colorado, June 2009. c⃝2009 Association for Computational Linguistics An Intrinsic Stopping Criterion for Committee-Based Active Learning Fredrik Olsson SICS Box 1263 SE-164 29 Kista, Sweden fredrik.olsson@sics.se Katrin Tomanek Jena University Language & Information Engineering Lab Friedrich-Schiller-Universit¨at Jena F¨urstengraben 30, D-07743 Jena, Germany katrin.tomanek@uni-jena.de Abstract As supervised machine learning methods are increasingly used in language technology, the need for high-quality annotated language data becomes imminent. Active learning (AL) is a means to alleviate the burden of annotation. This paper addresses the problem of knowing when to stop the AL process without having the human annotator make an explicit deci- sion on the matter. We propose and evaluate an intrinsic criterion for committee-based AL of named entity recognizers. 1 Introduction With the increasing popularity of supervised ma- chine learning methods in language processing, the need for high-quality labeled text becomes immi- nent. On the one hand, the amount of readily avail- able texts is huge, while on the other hand the la- beling and creation of corpora based on such texts is tedious, error prone and expensive. Active learning (AL) is one way of approaching the challenge of classifier creation and data annota- tion. Examples of AL used in language engineering include named entity recognition (Shen et al., 2004; Tomanek et al., 2007), text categorization (Lewis and Gale, 1994; Hoi et al., 2006), part-of-speech tagging (Ringger et al., 2007), and parsing (Thomp- son et al., 1999; Becker and Osborne, 2005). AL is a supervised machine learning technique in which the learner is in control of the data used for learning – the control is used to query an oracle, typ- ically a human, for the correct label of the unlabeled training instances for which the classifier learned so far makes unreliable predictions. The AL process takes as input a set of labeled in- stances and a larger set of unlabeled instances, and produces a classifier and a relatively small set of newly labeled data. The overall goal is to obtain as good a classifier as possible, without having to mark-up and supply the learner with more than nec- essary data. The learning process aims at keeping the human annotation effort to a minimum, only ask- ing for advice where the training utility of the result of such a query is high. The approaches taken to AL in this paper are based on committees of classifiers with access to pools of data. Figure 1 outlines a prototypical committee-based AL loop. In this paper we focus on the question when AL-driven annotation should be stopped (Item 7 in Figure 1). Usually, the progress of AL is illustrated by means of a learning curve which depicts how the classifier’s performance changes as a result of in- creasingly more labeled training data being avail- able. A learning curve might be used to address the issue of knowing when to stop the learning pro- cess – once the curve has leveled out, that …

Content truncated to 500 words