An Analysis of Active Learning Strategies for Sequence Labeling Tasks

Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 1070–1079, Honolulu, October 2008. c⃝2008 Association for Computational Linguistics An Analysis of Active Learning Strategies for Sequence Labeling Tasks Burr Settles∗† ∗Dept. of Computer Sciences University of Wisconsin Madison, WI 53706, USA bsettles@cs.wisc.edu Mark Craven†∗ †Dept. of Biostatistics & Medical Informatics University of Wisconsin Madison, WI 53706, USA craven@biostat.wisc.edu Abstract Active learning is well-suited to many prob- lems in natural language processing, where unlabeled data may be abundant but annota- tion is slow and expensive. This paper aims to shed light on the best active learning ap- proaches for sequence labeling tasks such as information extraction and document segmen- tation. We survey previously used query selec- tion strategies for sequence models, and pro- pose several novel algorithms to address their shortcomings. We also conduct a large-scale empirical comparison using multiple corpora, which demonstrates that our proposed meth- ods advance the state of the art. 1 Introduction Traditional supervised learning algorithms use whatever labeled data is provided to induce a model. By contrast, active learning gives the learner a de- gree of control by allowing it to select which in- stances are labeled and added to the training set. A typical active learner begins with a small labeled set L, selects one or more informative query instances from a large unlabeled pool U, learns from these la- beled queries (which are then added to L), and re- peats. In this way, the learner aims to achieve high accuracy with as little labeling effort as possible. Thus, active learning can be valuable in domains where unlabeled data are readily available, but ob- taining training labels is expensive. Such is the case with many sequence labeling tasks in natural language domains. For example, part-of-speech tagging (Seung et al., 1992; Lafferty et al., 2001), information extraction (Scheffer et al., 2001; Sang and DeMeulder, 2003; Kim et al., 2004), and document segmentation (Carvalho and Cohen, 2004) are all typically treated as sequence labeling problems. The source data for these tasks (i.e., text documents in electronic form) are often easily ob- tained. However, due to the nature of sequence la- beling tasks, annotating these texts can be rather te- dious and time-consuming, making active learning an attractive technique. While there has been much work on active learn- ing for classification (Cohn et al., 1994; McCallum and Nigam, 1998; Zhang and Oles, 2000; Zhu et al., 2003), active learning for sequence labeling has received considerably less attention. A few meth- ods have been proposed, based mostly on the con- ventions of uncertainty sampling, where the learner queries the instance about which it has the least cer- tainty (Scheffer et al., 2001; Culotta and McCallum, 2005; Kim et al., 2006), or query-by-committee, where a “committee” of models selects the instance about which its members most disagree (Dagan and Engelson, 1995). We provide more detail on these and the new strategies we propose in Section 3. The comparative effectiveness of these ap- proaches, however, has not …

Content truncated to 500 words