Guiding Semi-Supervision with Constraint-Driven Learning

Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 280–287, Prague, Czech Republic, June 2007. c⃝2007 Association for Computational Linguistics Guiding Semi-Supervision with Constraint-Driven Learning Ming-Wei Chang Lev Ratinov Department of Computer ScienceDan Roth University of Illinois at Urbana-Champaign Urbana, IL 61801 {mchang21, ratinov2, danr}@uiuc.edu Abstract Over the last few years, two of the main research directions in machine learning of natural language processing have been the study of semi-supervised learning algo- rithms as a way to train classifiers when the labeled data is scarce, and the study of ways to exploit knowledge and global information in structured learning tasks. In this paper, we suggest a method for incorporating do- main knowledge in semi-supervised learn- ing algorithms. Our novel framework unifies and can exploit several kinds of task specific constraints. The experimental results pre- sented in the information extraction domain demonstrate that applying constraints helps the model to generate better feedback during learning, and hence the framework allows for high performance learning with signif- icantly less training data than was possible before on these tasks. 1 Introduction Natural Language Processing (NLP) systems typi- cally require large amounts of knowledge to achieve good performance. Acquiring labeled data is a dif- ficult and expensive task. Therefore, an increasing attention has been recently given to semi-supervised learning, where large amounts of unlabeled data are used to improve the models learned from a small training set (Collins and Singer, 1999; Thelen and Riloff, 2002). The hope is that semi-supervised or even unsupervised approaches, when given enough knowledge about the structure of the problem, will be competitive with the supervised models trained on large training sets. However, in the general case, semi-supervised approaches give mixed re- sults, and sometimes even degrade the model per- formance (Nigam et al., 2000). In many cases, im- proving semi-supervised models was done by seed- ing these models with domain information taken from dictionaries or ontology (Cohen and Sarawagi, 2004; Collins and Singer, 1999; Haghighi and Klein, 2006; Thelen and Riloff, 2002). On the other hand, in the supervised setting, it has been shown that incorporating domain and problem specific struc- tured information can result in substantial improve- ments (Toutanova et al., 2005; Roth and Yih, 2005). This paper proposes a novel constraints-based learning protocol for guiding semi-supervised learn- ing. We develop a formalism for constraints-based learning that unifies several kinds of constraints: unary, dictionary based and n-ary constraints, which encode structural information and interdependencies among possible labels. One advantage of our for- malism is that it allows capturing different levels of constraint violation. Our protocol can be used in the presence of any learning model, including those that acquire additional statistical constraints from observed data while learning (see Section 5. In the experimental part of this paper we use HMMs as the underlying model, and exhibit significant reduction in the number of training examples required in two information extraction problems. As is often the case in semi-supervised learning, the algorithm can be viewed as a …

Content truncated to 500 words