Formalism-Independent Parser Evaluation with CCG and DepBank

Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 248–255, Prague, Czech Republic, June 2007. c⃝2007 Association for Computational Linguistics Formalism-Independent Parser Evaluation with CCG and DepBank Stephen Clark Oxford University Computing Laboratory Wolfson Building, Parks Road Oxford, OX1 3QD, UK stephen.clark@comlab.ox.ac.uk James R. Curran School of Information Technologies University of Sydney NSW 2006, Australia james@it.usyd.edu.au Abstract A key question facing the parsing commu- nity is how to compare parsers which use different grammar formalisms and produce different output. Evaluating a parser on the same resource used to create it can lead to non-comparable accuracy scores and an over-optimistic view of parser performance. In this paper we evaluate a CCG parser on DepBank, and demonstrate the difficulties in converting the parser output into Dep- Bank grammatical relations. In addition we present a method for measuring the effec- tiveness of the conversion, which provides an upper bound on parsing accuracy. The CCG parser obtains an F-score of 81.9% on labelled dependencies, against an upper bound of 84.8%. We compare the CCG parser against the RASP parser, outperform- ing RASP by over 5% overall and on the ma- jority of dependency types. 1 Introduction Parsers have been developed for a variety of gram- mar formalisms, for example HPSG (Toutanova et al., 2002; Malouf and van Noord, 2004), LFG (Ka- plan et al., 2004; Cahill et al., 2004), TAG (Sarkar and Joshi, 2003), CCG (Hockenmaier and Steed- man, 2002; Clark and Curran, 2004b), and variants of phrase-structure grammar (Briscoe et al., 2006), including the phrase-structure grammar implicit in the Penn Treebank (Collins, 2003; Charniak, 2000). Different parsers produce different output, for ex- ample phrase structure trees (Collins, 2003), depen- dency trees (Nivre and Scholz, 2004), grammati- cal relations (Briscoe et al., 2006), and formalism- specific dependencies (Clark and Curran, 2004b). This variety of formalisms and output creates a chal- lenge for parser evaluation. The majority of parser evaluations have used test sets drawn from the same resource used to develop the parser. This allows the many parsers based on the Penn Treebank, for example, to be meaningfully compared. However, there are two drawbacks to this approach. First, parser evaluations using different resources cannot be compared; for example, the Par- seval scores obtained by Penn Treebank parsers can- not be compared with the dependency F-scores ob- tained by evaluating on the Parc Dependency Bank. Second, using the same resource for development and testing can lead to an over-optimistic view of parser performance. In this paper we evaluate a CCG parser (Clark and Curran, 2004b) on the Briscoe and Carroll ver- sion of DepBank (Briscoe and Carroll, 2006). The CCG parser produces head-dependency relations, so evaluating the parser should simply be a matter of converting the CCG dependencies into those in Dep- Bank. Such conversions have been performed for other parsers, including parsers producing phrase structure output (Kaplan et al., 2004; Preiss, 2003). However, we found that performing such a conver- sion is a time-consuming and non-trivial task. The contributions of …

Content truncated to 500 words