A High-Performance Semi-Supervised Learning Method for Text Chunking

Proceedings of the 43rd Annual Meeting of the ACL, pages 1–9, Ann Arbor, June 2005. c⃝2005 Association for Computational Linguistics A High-Performance Semi-Supervised Learning Method for Text Chunking Rie Kubota Ando y Tong Zhang z IBM T.J. Watson Research Center Yorktown Heights, NY 10598, U.S.A. yrie1@us.ibm.com ztongz@us.ibm.com Abstract In machine learning, whether one can build a more accurate classifier by using unlabeled data (semi-supervised learning) is an important issue. Although a num- ber of semi-supervised methods have been proposed, their effectiveness on NLP tasks is not always clear. This paper presents a novel semi-supervised method that em- ploys a learning paradigm which we call structural learning. The idea is to find “what good classifiers are like” by learn- ing from thousands of automatically gen- erated auxiliary classification problems on unlabeled data. By doing so, the common predictive structure shared by the multiple classification problems can be discovered, which can then be used to improve perfor- mance on the target problem. The method produces performance higher than the pre- vious best results on CoNLL’00 syntac- tic chunking and CoNLL’03 named entity chunking (English and German). 1 Introduction In supervised learning applications, one can often find a large amount of unlabeled data without diffi- culty, while labeled data are costly to obtain. There- fore, a natural question is whether we can use unla- beled data to build a more accurate classifier, given the same amount of labeled data. This problem is often referred to as semi-supervised learning. Although a number of semi-supervised methods have been proposed, their effectiveness on NLP tasks is not always clear. For example, co-training (Blum and Mitchell, 1998) automatically bootstraps labels, and such labels are not necessarily reliable (Pierce and Cardie, 2001). A related idea is to use Expectation Maximization (EM) to impute la- bels. Although useful under some circumstances, when a relatively large amount of labeled data is available, the procedure often degrades performance (e.g. Merialdo (1994)). A number of bootstrap- ping methods have been proposed for NLP tasks (e.g. Yarowsky (1995), Collins and Singer (1999), Riloff and Jones (1999)). But these typically assume a very small amount of labeled data and have not been shown to improve state-of-the-art performance when a large amount of labeled data is available. Our goal has been to develop a general learning framework for reliably using unlabeled data to im- prove performance irrespective of the amount of la- beled data available. It is exactly this important and difficult problem that we tackle here. This paper presents a novel semi-supervised method that employs a learning framework called structural learning (Ando and Zhang, 2004), which seeks to discover shared predictive structures (i.e. what good classifiers for the task are like) through jointly learning multiple classification problems on unlabeled data. That is, we systematically create thousands of problems (called auxiliary problems) relevant to the target task using unlabeled data, and train classifiers from the automatically generated ‘training data’. We learn the commonality (or struc- ture) of such many classifiers relevant to the task, and …

Content truncated to 500 words