Using Grammatical Relations to Compare Parsers

Using Grammatical Relations to Compare Parsers Judita Preiss* Computer Laboratory University of Cambridge Judita.PreissAcl.cam.ac.uk Abstract We use the grammatical re- lations (GRs) described in Carroll et al. (1998) to compare a number of parsing algorithms A first ranking of the parsers is provided by comparing the extracted GRs to a gold standard GR anno- tation of 500 Susanne sentences: this required an implementation of GR extraction software for Penn Treebank style parsers. In addition, we perform an experiment using the extracted GRs as input to the Lappin and Leass (1994) anaphora resolution algorithm. This produces a second ranking of the parsers, and we investigate the number of errors that are caused by the incorrect GRs. 1 Introduction We investigate the usefulness of a grammatical relation (GR) evaluation method by using it to compare the performance of four full parsers and a GR finder based on a shallow parser. It is usually difficult to compare perfor- mance of different style parsers, as the out- put trees can vary in structure. In this pa- per, we use GRs to provide a common basis for comparing full and shallow parsers, and Penn Treebank and Susanne structures. To carry out this comparison, we implemented a GR extraction mechanism for Penn Treebank This work was supported by UK EPSRC project GR/N36462/93 'Robust Accurate Statistical Parsing'. parses. Evaluating parsers using GRs as op- posed to crossing brackets or labelled preci- sion/recall metrics can be argued to give a more robust measure of performance (Carroll et al., 1998), (Clark and Hockenmaier, 2002). The main novelty of this paper is the use of the Carroll et al's GR evaluation method to compare the Collins model 1 and model 2, and Charniak parsers. An initial evaluation is provided by compar- ing the extracted GRs to a gold standard GR annotation of 500 Susanne sentences due to Carroll et al. To gain insight into the strengths and weaknesses of the different parsers, we present a breakdown of the results for each type of GR. It is not clear whether the rank- ing produced from the gold standard evalu- ation is representative: there may be corpus effects for parsers not trained on Susanne, and real life applications may not reflect this ranking. We therefore perform an experi- ment using the extracted GRs as input to the Lappin and Leass (1994) anaphora resolution algorithm. This produces a second ranking of the parsers, and we investigate the number of errors that are caused by incorrect GRs. We describe the parsers and the GR finder in Section 2. We introduce GRs in Section 3 and briefly describe our GR extraction soft- ware for Penn Treebank style parses. The eval- uation, including a description of the evalua- tion corpus and performance results, is pre- sented in Section 4. The results are analyzed in Section 5 and a performance comparison in the context of anaphora resolution is presented in Section 6. We draw our conclusions in Sec- tion 7. 291 Parser Corpus LR LP CB OCB …

Content truncated to 500 words