Using Bracketed Parses to Evaluate a Grammar Checking Application

USING BRACKETED PARSES TO EVALUATE A GRAMMAR CHECKING APPLICATION Richard H. Wojcik, Philip Harrison, John Bremer Boeing Computer Services Research and Technology Division P.O. Box 24346, MS 7L--43 Seattle, WA 98124-2964 Internet: rwojcik@boeing.com, pharrison@boeing.com, jbremer@boeing.com Abstract We describe a method for evaluating a grammar checking application with hand-bracketed parses. A randomly-selected set of sentences was sub- mitted to a grammar checker in both bracketed and unbracketed formats. A comparison of the result- ing error reports illuminates the relationship be- tween the underlying performance of the parser- grammar system and the error critiques presented to the user. INTRODUCTION The recent development of broad-coverage natural language processing systems has stimu- lated work on the evaluation of the syntactic com- ponent of such systems, for purposes of basic eval- uation and improvement of system performance. Methods utilizing hand-bracketed corpora (such as the University of Pennsylvania Treebank) as a basis for evaluation metrics have been discussed in Black et al. (1991), Harrison et al. (1991), and Black et al. (1992). Three metrics discussed in those works were the Crossing Parenthesis Score (a count of the number of phrases in the machine produced parse which cross with one or more phrases in the hand parse), Recall (the percentage of phrases in the hand parse that are also in the ma- chine parse), and Precision (the percentage of phrases in the machine parse that are in the hand parse). We have developed a methodology for using hand-bracketed parses to examine both the inter- nal and external performance of a grammar checker. The internal performance refers to the behavior of the underlying system--i.e, the toke- nizer, parser, lexicon, and grammar. The external performance refers to the error critiques generated by the system. 1 Our evaluation methodology re- lies on three separate error reports generated from a corpus of randomly selected sentences: 1) a report based on unbracketed sentences, 2) a report based on optimally bracketed sentences with our current system, and 3) a report based on the opti- mal bracketings with the system modified to in- sure the same coverage as the unbracketed corpus. The bracketed report from the unmodified system tells us something about the coverage of our underlying system in its current state. The brack- eted report from the modified system tells us something about the external accuracy of the error reports presented to the user. Our underlying system uses a bottom-up, fun- ambiguity parser. Our error detection method relies on including grammar rules for parsing errorful sentences, with error critiques being gen- erated from the occurrence of an error rule in the parse. Error critiques are based on just one of all the possible parse trees that the system can find for a given sentence. Our major concern about the underlying system is whether the system has a cor- rect parse for the sentence in question. We are also concerned about the accuracy of the selected parse, but our current methodology does not directly address that issue, because correct error reports do not depend …

Content truncated to 500 words