Multi-Criteria-Based Strategy to Stop Active Learning for Data Annotation

Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008), pages 1129–1136 Manchester, August 2008 Multi-Criteria-based Strategy to Stop Active Learning for Data An- notation Jingbo Zhu Huizhen Wang Natural Language Processing Laboratory Northeastern University Shenyang, Liaoning, P.R.China 110004 zhujingbo@mail.neu.edu.cn wanghuizhen@mail.neu.edu.cn Eduard Hovy University of Southern California Information Sciences Institute 4676 Admiralty Way Marina del Rey, CA 90292-6695 hovy@isi.edu Abstract In this paper, we address the issue of de- ciding when to stop active learning for building a labeled training corpus. Firstly, this paper presents a new stopping crite- rion, classification-change, which con- siders the potential ability of each unla- beled example on changing decision boundaries. Secondly, a multi-criteria- based combination strategy is proposed to solve the problem of predefining an appropriate threshold for each confi- dence-based stopping criterion, such as max-confidence, min-error, and overall- uncertainty. Finally, we examine the ef- fectiveness of these stopping criteria on uncertainty sampling and heterogeneous uncertainty sampling for active learning. Experimental results show that these stopping criteria work well on evaluation data sets, and the combination strategies outperform individual criteria. 1 Introduction Creating a large labeled training corpus is very expensive and time-consuming in some real- world applications. For example, it is a crucial issue for automated word sense disambiguation task, because validations of sense definitions and sense-tagged data annotation have to be done by human experts, e.g. OntoNotes project (Hovy et al., 2006). Active learning aims to minimize the amount of human labeling effort by automatically select- ing the most informative unlabeled example for human annotation. In recent years active learning © 2008. Licensed under the Creative Commons Attri- bution-Noncommercial-Share Alike 3.0 Unported license (http://creativecommons.org/licenses/by-nc- sa/3.0/). Some rights reserved. has been widely studied in natural language processing (NLP) applications, such as word sense disambiguation (WSD) (Chen et al., 2006; Zhu and Hovy, 2007), text classification (TC) (Lewis and Gale, 1994; McCallum and Nigam, 1998a), named entity recognition (Shen et al., 2004), chunking (Ngai and Yarowsky, 2000), and statistical parsing (Tang et al., 2002). However, deciding when to stop active learn- ing is still an unsolved problem and seldom men- tioned issue in previous studies. Actually it is a very important practical issue in real-world ap- plications, because it obviously makes no sense to continue the active learning procedure until the whole unlabeled corpus has been labeled. The active learning process can be ended when the current classifier reaches the maximum effec- tiveness. In principle, how to learn a stopping criterion is a problem of estimation of classifier (i.e. learner) effectiveness during active learning (Lewis and Gale, 1994). In this paper, we address the issue of a stop- ping criterion for pool-based active learning with uncertainty sampling (Lewis and Gale, 1994), and propose a multi-criteria-based approach to determining when to stop active learning process. Firstly, this paper makes a comprehensive analy- sis on some confidence-based stopping criteria (Zhu and Hovy, 2007), including max- confidence, min-error and overall-uncertainty, then proposes a new stopping criterion, classifi- cation-change, which considers the potential ability of each unlabeled …

Content truncated to 500 words