Automatic Prediction of Parser Accuracy

Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 887–896, Honolulu, October 2008. c⃝2008 Association for Computational Linguistics Automatic Prediction of Parser Accuracy Sujith Ravi and Kevin Knight University of Southern California Information Sciences Institute Marina del Rey, California 90292 {sravi,knight}@isi.edu Radu Soricut Language Weaver, Inc. 4640 Admiralty Way, Suite 1210 Marina del Rey, California 90292 rsoricut@languageweaver.com Abstract Statistical parsers have become increasingly accurate, to the point where they are useful in many natural language applications. However, estimating parsing accuracy on a wide variety of domains and genres is still a challenge in the absence of gold-standard parse trees. In this paper, we propose a technique that au- tomatically takes into account certain charac- teristics of the domains of interest, and ac- curately predicts parser performance on data from these new domains. As a result, we have a cheap (no annotation involved) and effective recipe for measuring the performance of a sta- tistical parser on any given domain. 1 Introduction Statistical natural language parsers have recently become more accurate and more widely available. As a result, they are being used in a variety of applications, such as question answering (Herm- jakob, 2001), speech recognition (Chelba and Je- linek, 1998), language modeling (Roark, 2001), lan- guage generation (Soricut, 2006) and, most notably, machine translation (Charniak et al., 2003; Galley et al., 2004; Collins et al., 2005; Marcu et al., 2006; Huang et al., 2006; Avramidis and Koehn, 2008). These applications are employed on a wide range of domains and genres, and therefore the question of how accurate a parser is on the domain and genre of interest becomes acute. Ideally, one would want to have available a recipe for precisely answering this question: “given a parser and a particular domain of interest, how accurate are the parse trees produced?” The only recipe that is implicitly given in the large literature on parsing to date is to have human anno- tators build parse trees for a sample set from the do- main of interest, and consequently use them to com- pute a PARSEVAL (Black et al., 1991) score that is indicative of the intrinsic performance of the parser. Given the wide range of domains and genres for which NLP applications are of interest, combined with the high expertise required from human anno- tators to produce parse tree annotations, this recipe is, albeit precise, too expensive. The other recipe that is currently used on a large scale is to measure the performance of a parser on existing treebanks, such as WSJ (Marcus et al., 1993), and assume that the accuracy measure will carry over to the domains of interest. This recipe, albeit cheap, cannot provide any guarantee regarding the performance of a parser on a new domain, and, as experiments in this paper show, can give wrong indications regarding impor- tant decisions for the design of NLP systems that use a syntactic parser as an important component. This paper proposes another method for measur- ing the performance of a …

Content truncated to 500 words