Multi-Task Active Learning for Linguistic Annotations

Proceedings of ACL-08: HLT, pages 861–869, Columbus, Ohio, USA, June 2008. c⃝2008 Association for Computational Linguistics Multi-Task Active Learning for Linguistic Annotations Roi Reichart1∗ Katrin Tomanek2∗ Udo Hahn2 Ari Rappoport1 1Institute of Computer Science Hebrew University of Jerusalem, Israel {roiri|arir}@cs.huji.ac.il 2Jena University Language & Information Engineering (JULIE) Lab Friedrich-Schiller-Universit¨at Jena, Germany {katrin.tomanek|udo.hahn}@uni-jena.de Abstract We extend the classical single-task active learning (AL) approach. In the multi-task ac- tive learning (MTAL) paradigm, we select ex- amples for several annotation tasks rather than for a single one as usually done in the con- text of AL. We introduce two MTAL meta- protocols, alternating selection and rank com- bination, and propose a method to implement them in practice. We experiment with a two- task annotation scenario that includes named entity and syntactic parse tree annotations on three different corpora. MTAL outperforms random selection and a stronger baseline, one- sided example selection, in which one task is pursued using AL and the selected examples are provided also to the other task. 1 Introduction Supervised machine learning methods have success- fully been applied to many NLP tasks in the last few decades. These techniques have demonstrated their superiority over both hand-crafted rules and unsu- pervised learning approaches. However, they re- quire large amounts of labeled training data for every level of linguistic processing (e.g., POS tags, parse trees, or named entities). When, when domains and text genres change (e.g., moving from common- sense newspapers to scientific biology journal arti- cles), extensive retraining on newly supplied train- ing material is often required, since different do- mains may use different syntactic structures as well as different semantic classes (entities and relations). ∗ Both authors contributed equally to this work. Consequently, with an increasing coverage of a wide variety of domains in human language tech- nology (HLT) systems, we can expect a growing need for manual annotations to support many kinds of application-specific training data. Creating annotated data is extremely labor- intensive. The Active Learning (AL) paradigm (Cohn et al., 1996) offers a promising solution to deal with this bottleneck, by allowing the learning algorithm to control the selection of examples to be manually annotated such that the human label- ing effort be minimized. AL has been successfully applied already for a wide range of NLP tasks, in- cluding POS tagging (Engelson and Dagan, 1996), chunking (Ngai and Yarowsky, 2000), statistical parsing (Hwa, 2004), and named entity recognition (Tomanek et al., 2007). However, AL is designed in such a way that it se- lects examples for manual annotation with respect to a single learning algorithm or classifier. Under this AL annotation policy, one has to perform a separate annotation cycle for each classifier to be trained. In the following, we will refer to the annotations sup- plied for a classifier as the annotations for a single annotation task. Modern HLT systems often utilize annotations re- sulting from different tasks. For example, a machine translation system might use features extracted from parse trees and named entity annotations. For such an …

Content truncated to 500 words