Treebank Annotation Schemes and Parser Evaluation for German

Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pp. 630–639, Prague, June 2007. c⃝2007 Association for Computational Linguistics Treebank Annotation Schemes and Parser Evaluation for German Ines Rehbein NCLT School of Computing, DCU, Dublin, Ireland irehbein@computing.dcu.ie Josef van Genabith NCLT, School of Computing, DCU, Dublin, Ireland IBM Dublin Center for Advanced Studies josef@computing.dcu.ie Abstract Recent studies focussed on the question whether less-configurational languages like German are harder to parse than English, or whether the lower parsing scores are an artefact of treebank encoding schemes and data structures, as claimed by K¨ubler et al. (2006). This claim is based on the as- sumption that PARSEVAL metrics fully re- flect parse quality across treebank encoding schemes. In this paper we present new ex- periments to test this claim. We use the PARSEVAL metric, the Leaf-Ancestor met- ric as well as a dependency-based evalua- tion, and present novel approaches measur- ing the effect of controlled error insertion on treebank trees and parser output. We also provide extensive past-parsing cross- treebank conversion. The results of the ex- periments show that, contrary to K¨ubler et al. (2006), the question whether or not Ger- man is harder to parse than English remains undecided. 1 Introduction A long-standing and unresolved issue in the pars- ing literature is whether parsing less-configurational languages is harder than e.g. parsing English. Ger- man is a case in point. Results from Dubey and Keller (2003) suggest that state-of-the-art parsing scores for German are generally lower than those ob- tained for English, while recent results from K¨ubler et al. (2006) raise the possibility that this might be an artefact of particular encoding schemes and data structures of treebanks, which serve as training resources for probabilistic parsers. K¨ubler (2005) and Maier (2006) show that treebank annotation schemes have considerable influence on parsing re- sults. A comparison of unlexicalised PCFG pars- ing (K¨ubler, 2005) trained and evaluated on the Ger- man NEGRA (Skut et al., 1997) and the T¨uBa- D/Z (Telljohann et al., 2004) treebanks using LoPar (Schmid, 2000) shows a difference in parsing results of about 16%, using the PARSEVAL metric (Black et al., 1991). K¨ubler et al. (2006) conclude that, contrary to what had been assumed, German is not actually harder to parse than English, but that the NEGRA annotation scheme does not support opti- mal PCFG parsing performance. Despite being the standard metric for measuring PCFG parser performance, PARSEVAL has been criticised for not representing ’real’ parser quality (Carroll et al., 1998; Brisco et al., 2002; Sampson and Babarbczy, 2003). PARSEVALchecks label and wordspan identity in parser output compared to the original treebank trees. It neither weights results, differentiating between linguistically more or less severe errors, nor does it give credit to constituents where the syntactic categories have been recognised correctly but the phrase boundary is slightly wrong. With this in mind, we question the assumption that the PARSEVAL results for NEGRA and T¨uBa- D/Z reflect a real difference in quality between …

Content truncated to 500 words