Semi-Supervised Structured Output Learning Based on a Hybrid Generative and Discriminative Approach

Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pp. 791–800, Prague, June 2007. c⃝2007 Association for Computational Linguistics Semi-Supervised Structured Output Learning based on a Hybrid Generative and Discriminative Approach Jun Suzuki, Akinori Fujino and Hideki Isozaki NTT Communication Science Laboratories, NTT Corp. 2-4 Hikaridai, Seika-cho, Soraku-gun, Kyoto, 619-0237 Japan {jun, a.fujino, isozaki}@cslab.kecl.ntt.co.jp Abstract This paper proposes a framework for semi-supervised structured output learning (SOL), specifically for sequence labeling, based on a hybrid generative and discrim- inative approach. We define the objective function of our hybrid model, which is writ- ten in log-linear form, by discriminatively combining discriminative structured predic- tor(s) with generative model(s) that incor- porate unlabeled data. Then, unlabeled data is used in a generative manner to in- crease the sum of the discriminant functions for all outputs during the parameter estima- tion. Experiments on named entity recogni- tion (CoNLL-2003) and syntactic chunking (CoNLL-2000) data show that our hybrid model significantly outperforms the state- of-the-art performance obtained with super- vised SOL methods, such as conditional ran- dom fields (CRFs). 1 Introduction Structured output learning (SOL) methods, which attempt to optimize an interdependent output space globally, are important methodologies for certain natural language processing (NLP) tasks such as part-of-speech tagging, syntactic chunking (Chunk- ing) and named entity recognition (NER), which are also referred to as sequence labeling tasks. When we consider the nature of these sequence labeling tasks, a semi-supervised approach appears to be more nat- ural and appropriate. This is because the number of features and parameters typically become extremely large, and labeled examples can only sparsely cover the parameter space, even if thousands of labeled ex- amples are available. In fact, many attempts have re- cently been made to develop semi-supervised SOL methods (Zhu et al., 2003; Li and McCallum, 2005; Altun et al., 2005; Jiao et al., 2006; Brefeld and Scheffer, 2006). With the generative approach, we can easily in- corporate unlabeled data into probabilistic models with the help of expectation-maximization (EM) al- gorithms (Dempster et al., 1977). For example, the Baum-Welch algorithm is a well-known algorithm for training a hidden Markov model (HMM) of se- quence learning. Generally, with sequence learning tasks such as NER and Chunking, we cannot expect to obtain better performance than that obtained us- ing discriminative approaches in supervised learning settings. In contrast to the generative approach, with the discriminative approach, it is not obvious how un- labeled training data can be naturally incorporated into a discriminative training criterion. For ex- ample, the effect of unlabeled data will be elimi- nated from the objective function if the unlabeled data is directly used in traditional i.i.d. conditional- probability models. Nevertheless, several attempts have recently been made to incorporate unlabeled data in the discriminative approach. An approach based on pairwise similarities, which encourage nearby data points to have the same class label, has been proposed as a way of incorporating unlabeled data discriminatively (Zhu et al., 2003; Altun et al., 2005; Brefeld and …

Content truncated to 500 words