A Method for Stopping Active Learning Based on Stabilizing Predictions and the Need for User-Adjustable Stopping

Proceedings of the Thirteenth Conference on Computational Natural Language Learning (CoNLL), pages 39–47, Boulder, Colorado, June 2009. c⃝2009 Association for Computational Linguistics A Method for Stopping Active Learning Based on Stabilizing Predictions and the Need for User-Adjustable Stopping Michael Bloodgood∗ Human Language Technology Center of Excellence Johns Hopkins University Baltimore, MD 21211 USA bloodgood@jhu.edu K. Vijay-Shanker Computer and Information Sciences Department University of Delaware Newark, DE 19716 USA vijay@cis.udel.edu Abstract A survey of existing methods for stopping ac- tive learning (AL) reveals the needs for meth- ods that are: more widely applicable; more ag- gressive in saving annotations; and more sta- ble across changing datasets. A new method for stopping AL based on stabilizing predic- tions is presented that addresses these needs. Furthermore, stopping methods are required to handle a broad range of different annota- tion/performance tradeoff valuations. Despite this, the existing body of work is dominated by conservative methods with little (if any) at- tention paid to providing users with control over the behavior of stopping methods. The proposed method is shown to fill a gap in the level of aggressiveness available for stopping AL and supports providing users with control over stopping behavior. 1 Introduction The use of Active Learning (AL) to reduce NLP an- notation costs has generated considerable interest re- cently (e.g. (Bloodgood and Vijay-Shanker, 2009; Baldridge and Osborne, 2008; Zhu et al., 2008a)). To realize the savings in annotation efforts that AL enables, we must have a mechanism for knowing when to stop the annotation process. Figure 1 is intended to motivate the value of stop- ping at the right time. The x-axis measures the num- ber of human annotations that have been requested and ranges from 0 to 70,000. The y-axis measures ∗ This research was conducted while the first author was a PhD student at the University of Delaware. 0 1 2 3 4 5 6 7 x 10 4 65 70 75 80 85 90 Active Learning Curve (F Measure vs Number of Annotations) Number of Points for which Annotations Have Been Requested Performance (F Measure) stop point 1: stops too early; results in lower performing model stop point 2: good place to stop stop point 3: stops too late; wastes around 30,000 human annotations Figure 1: Hypothetical Active Learning Curve with hy- pothetical stopping points. performance in terms of F-Measure. As can be seen from the figure, the issue is that if we stop too early while useful generalizations are still being made, we wind up with a lower performing system but if we stop too late after all the useful generalizations have been made, we just wind up wasting human annota- tion effort. The terms aggressive and conservative will be used throughout the rest of this paper to describe the behavior of stopping methods. Conservative meth- ods tend to stop further to the right in Figure 1. They are conservative in the sense that they’re very careful not to risk losing significant amounts of F- measure, even if it means …

Content truncated to 500 words