Learning a Stopping Criterion for Active Learning for Word Sense Disambiguation and Text Classification

Learning a Stopping Criterion for Active Learning for Word Sense Disambiguation and Text Classification Jingbo Zhu Huizhen Wang Natural Language Processing Lab Northeastern University Shenyang, Liaoning, P.R.China, 110004 Zhujingbo@mail.neu.edu.cn wanghuizhen@mail.neu.edu.cn Eduard Hovy University of Southern California Information Sciences Institute 4676 Admiralty Way Marina del Rey, CA 90292-6695 hovy@isi.edu Abstract In this paper, we address the problem of knowing when to stop the process of active learning. We propose a new statistical learning approach, called minimum expected error strategy, to defining a stopping criterion through estimation of the classifier’s expected error on future unlabeled examples in the active learning process. In experiments on active learning for word sense disambiguation and text classification tasks, experimental results show that the new proposed stopping criterion can reduce approximately 50% human labeling costs in word sense disambiguation with degradation of 0.5% average accuracy, and approximately 90% costs in text classification with degradation of 2% average accuracy. 1 Introduction Supervised learning models set their parameters using given labeled training data, and generally outperform unsupervised learning methods when trained on equal amount of training data. However, creating a large labeled training corpus is very expensive and time-consuming in some real-world cases such as word sense disambiguation (WSD). Active learning is a promising way to minimize the amount of human labeling effort by building an system that automatically selects the most informa- tive unlabeled example for human annotation at each annotation cycle. In recent years active learn- ing has attracted a lot of research interest, and has been studied in many natural language processing (NLP) tasks, such as text classification (TC) (Lewis and Gale, 1994; McCallum and Nigam, 1998), chunking (Ngai and Yarowsky, 2000), named entity recognition (NER) (Shen et al., 2004; Tomanek et al., 2007), part-of-speech tagging (Engelson and Dagan, 1999), information extraction (Thompson et al., 1999), statistical parsing (Steedman et al., 2003), and word sense disambiguation (Zhu and Hovy, 2007). Previous studies reported that active learning can help in reducing human labeling effort. With selective sampling techniques such as uncertainty sampling (Lewis and Gale, 1994) and committee- based sampling (McCallum and Nigam, 1998), the size of the training data can be significantly re- duced for text classification (Lewis and Gale, 1994; McCallum and Nigam, 1998), word sense disambiguation (Chen, et al. 2006; Zhu and Hovy, 2007), and named entity recognition (Shen et al., 2004; Tomanek et al., 2007) tasks. Interestingly, deciding when to stop active learning is an issue seldom mentioned issue in these studies. However, it is an important practical topic, since it obviously makes no sense to continue the active learning procedure until the whole corpus has been labeled. How to define an adequate stopping criterion remains an unsolved problem in active learning. In principle, this is a problem of estimation of classifier effectiveness (Lewis and Gale, 1994). However, in real-world applications, it is difficult to know when the classifier reaches its maximum effectiveness before all unlabeled examples have been annotated. And when the unlabeled data set becomes very large, full annotation is almost …

Content truncated to 500 words