Effective Self-Training for Parsing

Proceedings of the Human Language Technology Conference of the North American Chapter of the ACL, pages 152–159, New York, June 2006. c⃝2006 Association for Computational Linguistics Effective Self-Training for Parsing David McClosky, Eugene Charniak, and Mark Johnson Brown Laboratory for Linguistic Information Processing (BLLIP) Brown University Providence, RI 02912 {dmcc|ec|mj}@cs.brown.edu Abstract We present a simple, but surprisingly ef- fective, method of self-training a two- phase parser-reranker system using read- ily available unlabeled data. We show that this type of bootstrapping is possible for parsing when the bootstrapped parses are processed by a discriminative reranker. Our improved model achieves an f-score of 92.1%, an absolute 1.1% improvement (12% error reduction) over the previous best result for Wall Street Journal parsing. Finally, we provide some analysis to bet- ter understand the phenomenon. 1 Introduction In parsing, we attempt to uncover the syntactic struc- ture from a string of words. Much of the challenge of this lies in extracting the appropriate parsing decisions from textual examples. Given sufficient labelled data, there are several “supervised” tech- niques of training high-performance parsers (Char- niak and Johnson, 2005; Collins, 2000; Henderson, 2004). Other methods are “semi-supervised” where they use some labelled data to annotate unlabeled data. Examples of this include self-training (Char- niak, 1997) and co-training (Blum and Mitchell, 1998; Steedman et al., 2003). Finally, there are “un- supervised” strategies where no data is labeled and all annotations (including the grammar itself) must be discovered (Klein and Manning, 2002). Semi-supervised and unsupervised methods are important because good labeled data is expensive, whereas there is no shortage of unlabeled data. While some domain-language pairs have quite a bit of labelled data (e.g. news text in English), many other categories are not as fortunate. Less unsuper- vised methods are more likely to be portable to these new domains, since they do not rely as much on ex- isting annotations. 2 Previous work A simple method of incorporating unlabeled data into a new model is self-training. In self-training, the existing model first labels unlabeled data. The newly labeled data is then treated as truth and com- bined with the actual labeled data to train a new model. This process can be iterated over different sets of unlabeled data if desired. It is not surprising that self-training is not normally effective: Charniak (1997) and Steedman et al. (2003) report either mi- nor improvements or significant damage from using self-training for parsing. Clark et al. (2003) applies self-training to POS-tagging and reports the same outcomes. One would assume that errors in the orig- inal model would be amplified in the new model. Parser adaptation can be framed as a semi- supervised or unsupervised learning problem. In parser adaptation, one is given annotated training data from a source domain and unannotated data from a target. In some cases, some annotated data from the target domain is available as well. The goal is to use the various data sets to produce a model that accurately parses the target domain data despite seeing little …

Content truncated to 500 words