Multi-Criteria-based Active Learning for Named Entity Recognition

Multi-Criteria-based Active Learning for Named Entity Recognition Dan Shen†‡1 Jie Zhang†‡ Jian Su† Guodong Zhou† Chew-Lim Tan‡ † Institute for Infocomm Technology 21 Heng Mui Keng Terrace Singapore 119613 ‡ Department of Computer Science National University of Singapore 3 Science Drive 2, Singapore 117543 {shendan,zhangjie,sujian,zhougd}@i2r.a-star.edu.sg {shendan,zhangjie,tancl}@comp.nus.edu.sg 1 Current address of the first author: Universität des Saarlandes, Computational Linguistics Dept., 66041 Saarbrü cken, Germany dshen@coli.uni-sb.de Abstract In this paper, we propose a multi-criteria- based active learning approach and effec- tively apply it to named entity recognition. Active learning targets to minimize the human annotation efforts by selecting ex- amples for labeling. To maximize the con- tribution of the selected examples, we consider the multiple criteria: informative- ness, representativeness and diversity and propose measures to quantify them. More comprehensively, we incorporate all the criteria using two selection strategies, both of which result in less labeling cost than single-criterion-based method. The results of the named entity recognition in both MUC-6 and GENIA show that the labeling cost can be reduced by at least 80% with- out degrading the performance. 1 Introduction In the machine learning approaches of natural lan- guage processing (NLP), models are generally trained on large annotated corpus. However, anno- tating such corpus is expensive and time- consuming, which makes it difficult to adapt an existing model to a new domain. In order to over- come this difficulty, active learning (sample selec- tion) has been studied in more and more NLP applications such as POS tagging (Engelson and Dagan 1999), information extraction (Thompson et al. 1999), text classification (Lewis and Catlett 1994; McCallum and Nigam 1998; Schohn and Cohn 2000; Tong and Koller 2000; Brinker 2003), statistical parsing (Thompson et al. 1999; Tang et al. 2002; Steedman et al. 2003), noun phrase chunking (Ngai and Yarowsky 2000), etc. Active learning is based on the assumption that a small number of annotated examples and a large number of unannotated examples are available. This assumption is valid in most NLP tasks. Dif- ferent from supervised learning in which the entire corpus are labeled manually, active learning is to select the most useful example for labeling and add the labeled example to training set to retrain model. This procedure is repeated until the model achieves a certain level of performance. Practically, a batch of examples are selected at a time, called batched- based sample selection (Lewis and Catlett 1994) since it is time consuming to retrain the model if only one new example is added to the training set. Many existing work in the area focus on two ap- proaches: certainty-based methods (Thompson et al. 1999; Tang et al. 2002; Schohn and Cohn 2000; Tong and Koller 2000; Brinker 2003) and commit- tee-based methods (McCallum and Nigam 1998; Engelson and Dagan 1999; Ngai and Yarowsky 2000) to select the most informative examples for which the current model are most uncertain. Being the first piece of work on active learning for name entity recognition (NER) task, we target to minimize the human annotation efforts yet …

Content truncated to 500 words