Measures and Models for Phrase Recognition

MEASURES AND MODELS FOR PHRASE RECOGNITION Steven Abney Bell Communications Research 445 South Street Morristown, NJ 07960 ABSTRACT I present an entropy measure for evaluating parser performance. The measure is fine-grained, and permits us to evaluate performance at the level of individual phrases. The parsing problem is characterized as statistically approximating the Penn Treebank annotations. I consider a series of models to "calibrate" the measure by determining what scores can be achieved using the most obvious kinds of information. I also relate the entropy measure to measures of recall/precision and grammar coverage. 1. INTRODUCTION Entropy measures of parser performance have focussed on the parser's contribution to word prediction. This is appropriate for evaluating a parser as a language model for speech recognition, but it is less appropriate for evaluating how well a parser does at parsing. I would like to present an entropy measure for phrase recognition, along with closely-related measures of precision and recall. I consider a seres of models, in order to establish a baseline for performance, and to give some sense of what parts of the problem are hardest, and what kinds of information contribute most to a solution. Specifically, I consider the problem of recognizing chunks (Abney 1991)--non-recursive pieces of major-category phrases, omitting post-head complements and modifiers. Chunks correspond to prosodic phrases (Abney 1992) and can be assembled into complete parse trees by adding head- head dependencies. 2. THE PARSING PROBLEM Parsing is usually characterized as the problem of recovering parse trees for sentences, given a grammar that defines the mapping of sentences to parse-trees. However, I wish to characterize the problem without assuming a grammar, for two reasons. First, we cannot assume a grammar for unrestricted English. For unrestricted English, failure of coverage will be a significant problem for any grammar, and we would like a measure of performance that treats failure of coverage and failures within the grammar uniformly. Second, I am particularly interested in parsers like Fidditch (Hindle 1983) and Cass (Abney 1990) that avoid search by relying on highly reliable patterns for recognizing individual phrases. Such parsers may need to consider competing patterns when scoring a given pattern--for example, Cass relies heavily on a preference for the pattern that matches the longest prefix of the input. Such cross- pattern dependencies cannot be expressed within, for example, a stochastic context-free grammar (SCFG). Hence I am interested in a more general evaluation framework, one that subsumes both Fidditch/Cass-style parsers and SCFG parsing. Instead of assuming a grammar, I take the Penn Treebank (Marcus & Santorini 1991) to provide a representative sample of English, viewed as a function from sentences to parse trees. A parser's task is to statistically approximate that function. We can measure the (in)accuracy of the parser by the amount of additional information we must provide in order to specify the correct (Treebank) parse for a sentence, given the output of the parser. This is the entropy of the corpus given the parser, and approaches zero as the parser approaches perfect emulation …

Content truncated to 500 words