Towards a More Careful Evaluation of Broad Coverage Parsing Systems

Towards a More Careful Evaluation of Broad Coverage Parsing Systems Wide It. Hogenhout and Yuji Matsumoto Nara Institute of Science and Technology 8916-5 Takayama, Ikoma Nara 630-01, Japan { marc- h, matsu } @is. aist- nara. ac.jp Abstract Since treebanks have become available to researchers a wide variety of tech- niques has been used to make broad cov- erage parsing systems. This makes quan- titative evaluation very important, but the current evaluation methods have a number of drawbacks such as arbitrary choices in the treebank and the difficulty in measuring statistical significance. We suggest a more detailed method for test- ing a parsing system using constituent boundaries, with a number of measures that give more information than current measures, and evaluate the quality of the test. We also show that statistical signif- icance cannot be calculated in a straight- forward way, and suggest a calculation method for the case of Bracket Recall. 1 Introduction During the last few years large treebanks have be- come available to many researchers, which has re- sulted in researches applying a range of new tech- niques for parsing systems. Most of the meth- ods that are being suggested include some kind of Machine Learning, such as history based gram- mars and decision tree models (Black et al., 1993; Magerman, 1995), training or inducing statisti- cal grammars (Black, Garside and Leech, 1993; Pereira and Schabes, 1992; Schabes et al., 1993), or other techniques (Bod, 1993). Consequently, syntactical analysis has become an area with a wide variety of (a) algorithms and methods for learning and parsing, and (b) type of information used for learning and parsing (some- times referred to as feature set). These meth- ods only could become popular through evalua- tion methods for parsing systems, such as Bracket Accuracy, Bracket Recall, Sentence Accuracy and Viterbi Score. Some of them were introduced in (Black et al., 1991; Harrison et M., 1991). These evaluation metrics have a number of problems, and in this paper we argue that they need to be reconsidered, and give a number of suggestions either to overcome those problems or to gain a better understanding of those prob- lems. Particular problems we look at are arbi- trary choices in the treebank, errors in the tree- bank, types of errors made by parsers, and the statistical significance of differences in test scores by parsers. 2 Problems with Evaluation Metrics Until now a number of problems with evaluation have been pointed out. One well known prob- lem is that measures based only on the absence of crossing errors on sentence level, such as Sen- tence Accuracy and Viterbi Consistency, are not usable for parsing systems that apply a partial bracketing, since a sparse bracketing improves the score. For example (Lin, 1995) discusses some other problems, but suggests an alternative that is difficult to apply. It is based on transferring constituency trees to dependency trees, but that introduces many ad hoc choices, and treebanks with dependency trees are hardly available. Also, a treebank usually contains arbitrary …

Content truncated to 500 words