Generalized Expectation Criteria for Semi-Supervised Learning of Conditional Random Fields

Proceedings of ACL-08: HLT, pages 870–878, Columbus, Ohio, USA, June 2008. c⃝2008 Association for Computational Linguistics Generalized Expectation Criteria for Semi-Supervised Learning of Conditional Random Fields Gideon S. Mann Google Inc. 76 Ninth Avenue New York, NY 10011 Andrew McCallum Department of Computer Science University of Massachusetts 140 Governors Drive Amherst, MA 01003 Abstract This paper presents a semi-supervised train- ing method for linear-chain conditional ran- dom fields that makes use of labeled features rather than labeled instances. This is accom- plished by using generalized expectation cri- teria to express a preference for parameter set- tings in which the model’s distribution on un- labeled data matches a target distribution. We induce target conditional probability distribu- tions of labels given features from both anno- tated feature occurrences in context and ad- hoc feature majority label assignment. The use of generalized expectation criteria allows for a dramatic reduction in annotation time by shifting from traditional instance-labeling to feature-labeling, and the methods presented outperform traditional CRF training and other semi-supervised methods when limited human effort is available. 1 Introduction A significant barrier to applying machine learning to new real world domains is the cost of obtaining the necessary training data. To address this prob- lem, work over the past several years has explored semi-supervised or unsupervised approaches to the same problems, seeking to improve accuracy with the addition of lower cost unlabeled data. Tradi- tional approaches to semi-supervised learning are applied to cases in which there is a small amount of fully labeled data and a much larger amount of un- labeled data, presumably from the same data source. For example, EM (Nigam et al., 1998), transduc- tive SVMs (Joachims, 1999), entropy regularization (Grandvalet and Bengio, 2004), and graph-based address : *number* oak avenue rent $ ADDRESS ADDRESS ADDRESS ADDRESS ADDRESS RENT RENT Traditional Full Instance Labeling ADDRESS address : *number* oak avenue rent $ .... CONTACT .. ( please include the address of this rental ) ADDRESS ... pm . address : *number* marie street sausalito ... ADDRESS .. laundry . address : *number* macarthur blvd .... Feature Labeling Conditional Distribution of Labels Given Word=address ADDRESS CONTACT Figure 1: Top: Traditional instance-labeling in which se- quences of contiguous tokens are annotated as to their correct label. Bottom: Feature-labeling in which non- contiguous feature occurrences in context are labeled for the purpose of deriving a conditional probability distribu- tion of labels given a particular feature. methods (Zhu and Ghahramani, 2002; Szummer and Jaakkola, 2002) have all been applied to a limited amount of fully labeled data in conjunction with un- labeled data to improve the accuracy of a classifier. In this paper, we explore an alternative approach in which, instead of fully labeled instances, the learner has access to labeled features. These fea- tures can often be labeled at a lower-cost to the hu- man annotator than labeling entire instances, which may require annotating the multiple sub-parts of a sequence structure or tree. Features can be labeled either by specifying the majority label …

Content truncated to 500 words