Self-Training for Enhancement and Domain Adaptation of Statistical Parsers Trained on Small Datasets

Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 616–623, Prague, Czech Republic, June 2007. c⃝2007 Association for Computational Linguistics Self-Training for Enhancement and Domain Adaptation of Statistical Parsers Trained on Small Datasets Roi Reichart ICNC Hebrew University of Jerusalem roiri@cs.huji.ac.il Ari Rappoport Institute of Computer Science Hebrew University of Jerusalem arir@cs.huji.ac.il Abstract Creating large amounts of annotated data to train statistical PCFG parsers is expensive, and the performance of such parsers declines when training and test data are taken from different domains. In this paper we use self- training in order to improve the quality of a parser and to adapt it to a different do- main, using only small amounts of manually annotated seed data. We report significant improvement both when the seed and test data are in the same domain and in the out- of-domain adaptation scenario. In particu- lar, we achieve 50% reduction in annotation cost for the in-domain case, yielding an im- provement of 66% over previous work, and a 20-33% reduction for the domain adaptation case. This is the first time that self-training with small labeled datasets is applied suc- cessfully to these tasks. We were also able to formulate a characterization of when self- training is valuable. 1 Introduction State of the art statistical parsers (Collins, 1999; Charniak, 2000; Koo and Collins, 2005; Charniak and Johnson, 2005) are trained on manually anno- tated treebanks that are highly expensive to create. Furthermore, the performance of these parsers de- creases as the distance between the genres of their training and test data increases. Therefore, enhanc- ing the performance of parsers when trained on small manually annotated datasets is of great impor- tance, both when the seed and test data are taken from the same domain (the in-domain scenario) and when they are taken from different domains (the out- of-domain or parser adaptation scenario). Since the problem is the expense in manual annotation, we de- fine ‘small’ to be 100-2,000 sentences, which are the sizes of sentence sets that can be manually annotated by constituent structure in a few hours1. Self-training is a method for using unannotated data when training supervised models. The model is first trained using manually annotated (‘seed’) data, then the model is used to automatically annotate a pool of unannotated (‘self-training’) data, and then the manually and automatically annotated datasets are combined to create the training data for the fi- nal model. Self-training of parsers trained on small datasets is of enormous potential practical impor- tance, due to the huge amounts of unannotated data that are becoming available today and to the high cost of manual annotation. In this paper we use self-training to enhance the performance of a generative statistical PCFG parser (Collins, 1999) for both the in-domain and the parser adaptation scenarios, using only small amounts of manually annotated data. We perform four experi- ments, examining all combinations of in-domain and out-of-domain seed and self-training data. Our results show that self-training is of substantial benefit for …

Content truncated to 500 words