An Efficient Active Learning Framework for New Relation Types

International Joint Conference on Natural Language Processing, pages 692–698, Nagoya, Japan, 14-18 October 2013. An Efficient Active Learning Framework for New Relation Types Lisheng Fu Computer Science Department New York University lf1099@nyu.edu Ralph Grishman Computer Science Department New York University grishman@cs.nyu.edu Abstract Supervised training of models for semantic relation extraction has yielded good perfor- mance, but at substantial cost for the annota- tion of large training corpora. Active learn- ing strategies can greatly reduce this annota- tion cost. We present an efficient active learning framework that starts from a better balance between positive and negative sam- ples, and boosts training efficiency by inter- leaving self-training and co-testing. We also studied the reduction of annotation cost by enforcing argument type constraints. Experi- ments show a substantial speed-up by com- parison to the previous state-of-the-art pure co-testing active learning framework. We ob- tain reasonable performance with only 150 labels for individual ACE 2004 relation types. 1 Introduction Relation extraction aims to discover the semantic relationship, if any, between a pair of entities in text. This structured information can be used to build higher-level applications such as question answering and other text mining applications. Relation extraction was intensively studied as part of the multi-site ACE [Automatic Content Extraction] evaluations conducted in 2003, 2004, and 2005. For 2004, six major relation types were defined. Each relation mention takes two entity mention arguments in the same sentence. In annotating text, each entity mention pair with- in one sentence will be labeled if it involves one of the relation types. As part of ACE, substantial hand-annotated corpora marked with entities and relations were produced. For example, the ACE 2004 corpus had in total about 5,000 relation in- stances (and about 45,000 same-sentence entity pairs not bearing one of these relations). These large training corpora stimulated research on the supervised training of relation extractors, with considerable success: the best systems, when given hand-tagged entities, correctly identify and classify relations with an F score above 70% (Jiang and Zhai 2007). Although supervised methods were effective, annotating a corpus of this size is too expensive in practice to serve as a model for developing new extractors: it requires consideration of 50K instances, of which only a small portion involve the target relation type. In consequence, most research has focused on reducing the annotation cost through semi-supervised learning methods such as bootstrapping systems. However, with limited labeled data, those semi-supervised sys- tems failed to come close to the supervised level of performance. Their performance also varies with the distribution of seeds. Recent studies have proposed new ways of re- ducing the annotation cost by using active learn- ing. The advantage of active learning is that it can achieve reasonable performance, and even performance comparable to the supervised ver- sion, with few labeled examples, due to its ability to selectively sample unlabeled data for annota- tion. To further reduce the annotation cost and pro- vide an efficient framework for rapidly develop- ing relation extraction models, we combine ac- tive learning with …

Content truncated to 500 words