Minimizing Manual Annotation Cost in Supervised Training from Corpora

Minimizing Manual Annotation Cost In Supervised Training From Corpora Sean P. Engelson and Ido Dagan Department of Mathematics and Computer Science Bar-Ilan University 52900 Ramat Gan, Israel {engelson, dagan}@bimacs, cs. biu. ac. il Abstract Corpus-based methods for natural lan- guage processing often use supervised training, requiring expensive manual an- notation of training corpora. This paper investigates methods for reducing annota- tion cost by sample selection. In this ap- proach, during training the learning pro- gram examines many unlabeled examples and selects for labeling (annotation) only those that are most informative at each stage. This avoids redundantly annotating examples that contribute little new infor- mation. This paper extends our previous work on committee-based sample selection for probabilistic classifiers. We describe a family of methods for committee-based sample selection, and report experimental results for the task of stochastic part-of- speech tagging. We find that all variants achieve a significant reduction in annota- tion cost, though their computational effi- ciency differs. In particular, the simplest method, which has no parameters to tune, gives excellent results. We also show that sample selection yields a significant reduc- tion in the size of the model used by the tagger. 1 Introduction Many corpus-based methods for natural language processing (NLP) are based on supervised training-- acquiring information from a manually annotated corpus. Therefore, reducing annotation cost is an important research goal for statistical NLP. The ul- timate reduction in annotation cost is achieved by unsupervised training methods, which do not require an annotated corpus at all (Kupiec, 1992; Merialdo, 1994; Elworthy, 1994). It has been shown, how- ever, that some supervised training prior to the un- supervised phase is often beneficial. Indeed, fully unsupervised training may not be feasible for cer- tain tasks. This paper investigates an approach for optimizing the supervised training (learning) phase, which reduces the annotation effort required to achieve a desired level of accuracy of the trained model. In this paper, we investigate and extend the committee-based sample selection approach to min- imizing training cost (Dagan and Engelson, 1995). When using sample selection, a learning program ex- amines many unlabeled (not annotated) examples, selecting for labeling only those that are most in- formative for the learner at each stage of training (Seung, Opper, and Sompolinsky, 1992; Freund et al., 1993; Lewis and Gale, 1994; Cohn, Atlas, and Ladner, 1994). This avoids redundantly annotating many examples that contribute roughly the same in- formation to the learner. Our work focuses on sample selection for training probabilistic classifiers. In statistical NLP, prob- abilistic classifiers are often used to select a pre- ferred analysis of the linguistic structure of a text (for example, its syntactic structure (Black et al., 1993), word categories (Church, 1988), or word senses (Gale, Church, and Yarowsky, 1993)). As a representative task for probabilistic classification in NLP, we experiment in this paper with sample se- lection for the popular and well-understood method of stochastic part-of-speech tagging using Hidden Markov Models. We first review the basic approach of committee- based sample selection and its …

Content truncated to 500 words