Semi-Supervised Conditional Random Fields for Improved Sequence Segmentation and Labeling

Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the ACL, pages 209–216, Sydney, July 2006. c⃝2006 Association for Computational Linguistics Semi-Supervised Conditional Random Fields for Improved Sequence Segmentation and Labeling Feng Jiao University of Waterloo Shaojun Wang Chi-Hoon Lee Russell Greiner Dale Schuurmans University of Alberta Abstract We present a new semi-supervised training procedure for conditional random fields (CRFs) that can be used to train sequence segmentors and labelers from a combina- tion of labeled and unlabeled training data. Our approach is based on extending the minimum entropy regularization frame- work to the structured prediction case, yielding a training objective that combines unlabeled conditional entropy with labeled conditional likelihood. Although the train- ing objective is no longer concave, it can still be used to improve an initial model (e.g. obtained from supervised training) by iterative ascent. We apply our new training algorithm to the problem of iden- tifying gene and protein mentions in bio- logical texts, and show that incorporating unlabeled data improves the performance of the supervised CRF in this case. 1 Introduction Semi-supervised learning is often touted as one of the most natural forms of training for language processing tasks, since unlabeled data is so plen- tiful whereas labeled data is usually quite limited or expensive to obtain. The attractiveness of semi- supervised learning for language tasks is further heightened by the fact that the models learned are large and complex, and generally even thousands of labeled examples can only sparsely cover the parameter space. Moreover, in complex structured prediction tasks, such as parsing or sequence mod- eling (part-of-speech tagging, word segmentation, named entity recognition, and so on), it is con- siderably more difficult to obtain labeled training data than for classification tasks (such as docu- ment classification), since hand-labeling individ- ual words and word boundaries is much harder than assigning text-level class labels. Many approaches have been proposed for semi- supervised learning in the past, including: genera- tive models (Castelli and Cover 1996; Cohen and Cozman 2006; Nigam et al. 2000), self-learning (Celeux and Govaert 1992; Yarowsky 1995), co- training (Blum and Mitchell 1998), information- theoretic regularization (Corduneanu and Jaakkola 2006; Grandvalet and Bengio 2004), and graph- based transductive methods (Zhou et al. 2004; Zhou et al. 2005; Zhu et al. 2003). Unfortu- nately, these techniques have been developed pri- marily for single class label classification prob- lems, or class label classification with a struc- tured input (Zhou et al. 2004; Zhou et al. 2005; Zhu et al. 2003). Although still highly desirable, semi-supervised learning for structured classifica- tion problems like sequence segmentation and la- beling have not been as widely studied as in the other semi-supervised settings mentioned above, with the sole exception of generative models. With generative models, it is natural to include unlabeled data using an expectation-maximization approach (Nigam et al. 2000). However, gener- ative models generally do not achieve the same accuracy as discriminatively trained models, and therefore it is preferable to focus on discriminative approaches. Unfortunately, …

Content truncated to 500 words