An Ensemble Method for Selection of High Quality Parses

Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 408–415, Prague, Czech Republic, June 2007. c⃝2007 Association for Computational Linguistics An Ensemble Method for Selection of High Quality Parses Roi Reichart ICNC Hebrew University of Jerusalem roiri@cs.huji.ac.il Ari Rappoport Institute of Computer Science Hebrew University of Jerusalem arir@cs.huji.ac.il Abstract While the average performance of statisti- cal parsers gradually improves, they still at- tach to many sentences annotations of rather low quality. The number of such sentences grows when the training and test data are taken from different domains, which is the case for major web applications such as in- formation retrieval and question answering. In this paper we present a Sample Ensem- ble Parse Assessment (SEPA) algorithm for detecting parse quality. We use a function of the agreement among several copies of a parser, each of which trained on a differ- ent sample from the training data, to assess parse quality. We experimented with both generative and reranking parsers (Collins, Charniak and Johnson respectively). We show superior results over several baselines, both when the training and test data are from the same domain and when they are from different domains. For a test setting used by previous work, we show an error reduction of 31% as opposed to their 20%. 1 Introduction Many algorithms for major NLP applications such as information extraction (IE) and question answer- ing (QA) utilize the output of statistical parsers (see (Yates et al., 2006)). While the average per- formance of statistical parsers gradually improves, the quality of many of the parses they produce is too low for applications. When the training and test data are taken from different domains (the parser adaptation scenario) the ratio of such low quality parses becomes even higher. Figure 1 demonstrates these phenomena for two leading models, Collins (1999) model 2, a generative model, and Charniak and Johnson (2005), a reranking model. The parser adaptation scenario is the rule rather than the excep- tion for QA and IE systems, because these usually operate over the highly variable Web, making it very difficult to create a representative corpus for manual annotation. Medium quality parses may seriously harm the performance of such systems. In this paper we address the problem of assess- ing parse quality, using a Sample Ensemble Parse Assessment (SEPA) algorithm. We use the level of agreement among several copies of a parser, each of which trained on a different sample from the training data, to predict the quality of a parse. The algorithm does not assume uniformity of training and test data, and is thus suitable to web-based applications such as QA and IE. Generative statistical parsers compute a probabil- ity p(a, s) for each sentence annotation, so the im- mediate technique that comes to mind for assess- ing parse quality is to simply use p(a, s). Another seemingly trivial method is to assume that shorter sentences would be parsed better than longer ones. However, these techniques produce results that are far from optimal. In …

Content truncated to 500 words