Comparative Parser Performance Analysis across Grammar Frameworks through Automatic Tree Conversion using Synchronous Grammars

Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008), pages 545–552 Manchester, August 2008 Comparative Parser Performance Analysis across Grammar Frameworks through Automatic Tree Conversion using Synchronous Grammars Takuya Matsuzaki 1 Jun’ichi Tsujii 1,2,3 1. Department of Computer Science, University of Tokyo, Japan 2. School of Computer Science, University of Manchester, UK 3. National Center for Text Mining, UK {matuzaki, tsujii}@is.s.u-tokyo.ac.jp Abstract This paper presents a methodology for the comparative performance analysis of the parsers developed for different grammar frameworks. For such a comparison, we need a common representation format of the parsing results since the representation of the parsing results depends on the gram- mar frameworks; hence they are not di- rectly comparable to each other. We first convert the parsing result to a shallow CFG analysis by using an automatic tree con- verter based on synchronous grammars. The use of such a shallow representation as a common format has the advantage of re- duced noise introduced by the conversion in comparison with the noise produced by the conversion to deeper representations. We compared an HPSG parser with sev- eral CFG parsers in our experiment and found that meaningful differences among the parsers’ performance can still be ob- served by such a shallow representation. 1 Introduction Recently, there have been advancement made in the parsing techniques for large-scale lexicalized grammars (Clark and Curran, 2004; Ninomiya et al., 2005; Ninomiya et al., 2007), and it have presumably been accelerated by the development of the semi-automatic acquisition techniques of large-scale lexicalized grammars from parsed cor- pora (Hockenmaier and Steedman, 2007; Miyao c⃝ 2008. Licensed under the Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported li- cense (http://creativecommons.org/licenses/by-nc-sa/3.0/). Some rights reserved. et al., 2005). In many of the studies on lexical- ized grammar parsing, the accuracy of the pars- ing results is evaluated in terms of the accuracy of the semantic representations output by the parsers. Since the formalisms for the semantic representa- tion are different across the grammar frameworks, it has been difficult to directly compare the perfor- mance of the parsers developed for different gram- mar frameworks. Several researchers in the field of lexicalized grammar parsing have recently started to seek a common representation of parsing results across different grammar frameworks (Clark and Curran, 2007; Miyao et al., 2007). For example, Clark and Curran (2007) developed a set of mapping rules from the output of a Combinatorial Catego- rial grammar parser to the Grammatical Relations (GR) (Carroll et al., 1998). They found that the manual development of such mapping rules is not a trivial task; their mapping rules covered only 85% of the GRs in a GR-annotated corpus; i.e., 15% of the GRs in the corpus could not be covered by the mapping from the gold-standard CCG analyses of those sentences. We propose another method for the cross- framework performance analysis of the parsers wherein the output of parsers are first converted to a CFG tree. Specifically, we use CFG trees of the style used in the Penn Treebank (PTB) …

Content truncated to 500 words