An Experimental Comparison of Active Learning Strategies for Partially Labeled Sequences

Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 898–906, October 25-29, 2014, Doha, Qatar. c⃝2014 Association for Computational Linguistics An Experimental Comparison of Active Learning Strategies for Partially Labeled Sequences Diego Marcheggiani Istituto di Scienza e Tecnologie dell’Informazione Consiglio Nazionale delle Ricerche Pisa, Italy diego.marcheggiani@isti.cnr.it Thierry Arti`eres LIP6 Pierre et Marie Curie University Paris, France thierry.artieres@lip6.fr Abstract Active learning (AL) consists of asking human annotators to annotate automatically selected data that are assumed to bring the most bene- fit in the creation of a classifier. AL allows to learn accurate systems with much less anno- tated data than what is required by pure super- vised learning algorithms, hence limiting the tedious effort of annotating a large collection of data. We experimentally investigate the behav- ior of several AL strategies for sequence labeling tasks (in a partially-labeled sce- nario) tailored on Partially-Labeled Condi- tional Random Fields, on four sequence la- beling tasks: phrase chunking, part-of-speech tagging, named-entity recognition, and bio- entity recognition. 1 Introduction Today, the state-of-the-art methods in most natural lan- guage processing tasks are supervised machine learn- ing approaches. Their main problem lies in their need of large human-annotated training corpus, which re- quires a tedious and expensive work from domain ex- perts. The process of active learning (AL) employs one or more human annotators by asking them to label new samples which are supposed to be the most informa- tive in the creation of a new classifier. A classifier is incrementally retrained with all the data labeled by the annotator. AL has been demonstrated to work well and to produce accurate classifiers while saving much hu- man annotation effort. One critical issue is to define a measure of the informativeness which should reflect how much new information a new example would give in the learning of a new classifier once annotated. A lot of work has been done on the AL field in the past years (see (Settles, 2012) for an exhaustive overview). In particular, AL proved its usefulness in se- quence labeling tasks (Settles and Craven, 2008). Yet, researchers have always adopted as annotation unit an entire sequence (i.e., the annotator is asked to anno- tate the whole sequence) while it looks like it could be much more relevant to ask for labeling only small parts of it (e.g., the ones with highest ambiguity). A few works have investigated this idea. For instance, Wan- varie et al. (2011) proposed to use Partially-Labeled Conditional Random Fields (PL-CRFs) (Tsuboi et al., 2008), a semi-supervised variation of Conditional Ran- dom Fields (CRFs) (Lafferty et al., 2001) able to deal with partially-labeled sequences, thus enabling to adopt as annotation unit single tokens and still learning from full sequences. AL with partially labeled sequences has proven to be effective in substantially reducing the amount of annotated data with respect to common AL approaches (see (Wanvarie et al., 2011)). In this work we focus on AL strategies for partially labeled sequences adopting the single token as annota- …

Content truncated to 500 words