Stopping Criteria for Active Learning of Named Entity Recognition

Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008), pages 465–472 Manchester, August 2008 Stopping Criteria for Active Learning of Named Entity Recognition Florian Laws Institute for NLP Universit¨at Stuttgart fl@ifnlp.org Hinrich Sch¨utze Institute for NLP Universit¨at Stuttgart hs999@ifnlp.org Abstract Active learning is a proven method for re- ducing the cost of creating the training sets that are necessary for statistical NLP. How- ever, there has been little work on stopping criteria for active learning. An operational stopping criterion is necessary to be able to use active learning in NLP applications. We investigate three different stopping cri- teria for active learning of named entity recognition (NER) and show that one of them, gradient-based stopping, (i) reliably stops active learning, (ii) achieves near- optimal NER performance, (iii) and needs only about 20% as much training data as exhaustive labeling. 1 Introduction Supervised statistical learning methods are impor- tant and widely successful tools for natural lan- guage processing. These methods learn by esti- mating a statistical model on labeled training data. Often, these models require a large amount of training data that needs to be hand-annotated by human experts. This is time-consuming and ex- pensive. Active learning (AL) reduces this annota- tion effort by selecting unlabeled examples that are maximally informative for the statistical learning method and handing them to a human annotator for labeling. The statistical model is then updated with the newly gathered information. In this pa- per, we adopt the uncertainty sampling approach to AL (Lewis and Gale, 1994). Uncertainty sam- pling selects those examples in the pool as most in- c⃝ 2008. Licensed under the Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported li- cense (http://creativecommons.org/licenses/by-nc-sa/3.0/). Some rights reserved. formative for which the statistical classifier is least certain in its classification decision. While AL is an active area of research in NLP, the issue of determining when to stop the AL process has only recently come into focus (Zhu and Hovy, 2007; Vlachos, 2008). This is some- what surprising because the main purpose of ac- tive learning is to save on annotation effort; decid- ing on the point when enough data is annotated is crucial to fulfilling this goal. We investigate three different stopping criteria in this paper. First, a user of a classification system may want to set a minimum absolute performance for the system to be deployed. The standard way of assessing classifier performance uses a held-out la- beled test set. However, labeling a test set of suffi- cient size is contrary to the goal of minimizing an- notation effort and impractical in most real-world settings. We will show that the classifier can esti- mate its own performance using only an unlabeled reference set and propose to stop active learning if estimated performance reaches the threshold set by the user. The estimation is somewhat inaccu- rate, however, and we investigate possible reasons for estimation error. An alternative criterion is based on maximum possible performance. We will show that our per- formance estimation method supports stopping AL …

Content truncated to 500 words