All Fragments Count in Parser Evaluation

All Fragments Count in Parser Evaluation Jasmijn Bastings, Khalil Sima’an Institute for Logic, Language and Computation University of Amsterdam bastings@uva.nl, k.simaan@uva.nl Abstract PARSEVAL, the default paradigm for evaluating constituency parsers, calculates parsing success (Precision/Recall) as a function of the number of matching labeled brackets across the test set. Nodes in constituency trees, however, are connected together to reflect important linguistic relations such as predicate-argument and direct-dominance relations between categories. In this paper, we present FREVAL, a generalization of PARSEVAL, where the precision and recall are calculated not only for individual brackets, but also for co-occurring, connected brackets (i.e. fragments). FREVAL fragments precision (FLP) and recall (FLR) interpolate the match across the whole spectrum of fragment sizes ranging from those consisting of individual nodes (labeled brackets) to those consisting of full parse trees. We provide evidence that FREVAL is informative for inspecting relative parser performance by comparing a range of existing parsers. Keywords: fragments, parsing, evaluation 1. Motivation Current approaches to parsing usually employ a training treebank to learn a statistical parser. The goal of learning is to obtain a parser that can reproduce the test set tree- bank parses as accurately as possible. The rationale behind this is that the treebank parses themselves are products of trained human annotators and, hence, should serve as the gold standard. If indeed the ultimate goal of learning statistical parsers from treebanks is to obtain parsers that immitate human parsing capability as represented by a sample in a tree- bank, then parser evaluation should aim at measuring the amount match/mismatch between a parse produced by a parser and a parse produced by human annotation. It is crucial at this point to highlight the difference between this view of parser evaluation and a linguistically-oriented point of view: a linguistically-motivated parser evaluation fo- cuses on linguistically-relevant aspects of parse trees that are crucial for subsequent linguistic processing, e.g., de- pendency relations might be very important for semantic or other linguistic processing (cf. alternative linguistically relevant proposals (Sampson and Babarczy, 2003; Carroll et al., 1998)). The contrast between linguistic relevance and the statistical view of treebank parsing as a learning problem (with some cognitive relevance) is crucial, because parser output often has other practical uses besides serv- ing as mere input for subsequent linguistic processing, e.g., parsers may serve as target language models in machine translation systems. Consequently, to evaluate a statistical parser learned from a treebank we need a measure of similarity between its output parse and the human annotated parse in the test set. Such a measure of similarity between two tre could measure dif- ferent shared aspects between two trees. The PARSEVAL measures (Black et al., 1991) are currently the de facto stan- dard for evaluating (English) parser output. To calculate the Precision and Recall, the output trees of a parser are com- pared to a gold standard, i.e. human-annotated trees in a treebank. A well known treebank in this respect is the Penn Wall Street Journal treebank (Marcus et al., 1993). To fa- …

Content truncated to 500 words