Cross-Framework Evaluation for Statistical Parsing

Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 44–54, Avignon, France, April 23 - 27 2012. c⃝2012 Association for Computational Linguistics Cross-Framework Evaluation for Statistical Parsing Reut Tsarfaty Joakim Nivre Evelina Andersson Uppsala University, Box 635, 75126 Uppsala, Sweden tsarfaty@stp.lingfil.uu.se,{joakim.nivre,evelina.andersson}@lingfil.uu.se Abstract A serious bottleneck of comparative parser evaluation is the fact that different parsers subscribe to different formal frameworks and theoretical assumptions. Converting outputs from one framework to another is less than optimal as it easily introduces noise into the process. Here we present a principled protocol for evaluating parsing results across frameworks based on func- tion trees, tree generalization and edit dis- tance metrics. This extends a previously proposed framework for cross-theory eval- uation and allows us to compare a wider class of parsers. We demonstrate the useful- ness and language independence of our pro- cedure by evaluating constituency and de- pendency parsers on English and Swedish. 1 Introduction The goal of statistical parsers is to recover a for- mal representation of the grammatical relations that constitute the argument structure of natural language sentences. The argument structure en- compasses grammatical relationships between el- ements such as subject, predicate, object, etc., which are useful for further (e.g., semantic) pro- cessing. The parses yielded by different parsing frameworks typically obey different formal and theoretical assumptions concerning how to rep- resent the grammatical relationships in the data (Rambow, 2010). For example, grammatical rela- tions may be encoded on top of dependency arcs in a dependency tree (Mel’ˇcuk, 1988), they may decorate nodes in a phrase-structure tree (Marcus et al., 1993; Maamouri et al., 2004; Sima’an et al., 2001), or they may be read off of positions in a phrase-structure tree using hard-coded conver- sion procedures (de Marneffe et al., 2006). This diversity poses a challenge to cross-experimental parser evaluation, namely: How can we evaluate the performance of these different parsers relative to one another? Current evaluation practices assume a set of correctly annotated test data (or gold standard) for evaluation. Typically, every parser is eval- uated with respect to its own formal representa- tion type and the underlying theory which it was trained to recover. Therefore, numerical scores of parses across experiments are incomparable. When comparing parses that belong to different formal frameworks, the notion of a single gold standard becomes problematic, and there are two different questions we have to answer. First, what is an appropriate gold standard for cross-parser evaluation? And secondly, how can we alle- viate the differences between formal representa- tion types and theoretical assumptions in order to make our comparison sound – that is, to make sure that we are not comparing apples and oranges? A popular way to address this has been to pick one of the frameworks and convert all parser outputs to its formal type. When com- paring constituency-based and dependency-based parsers, for instance, the output of constituency parsers has often been converted to dependency structures prior to evaluation (Cer et al., 2010; Nivre et al., …

Content truncated to 500 words