Evaluation Tool for Rule-based Anaphora Resolution Methods

Evaluation tool for rule-based anaphora resolution methods Catalina Barbu School of Humanities, Languages and Social Sciences University of Wolverhampton Stafford Street Wolverhampton WV1 1SB United Kingdom c.barbu@wlv.ac.uk Ruslan Mitkov School of Humanities, Languages and Social Sciences University of Wolverhampton Stafford Street Wolverhampton WV1 1SB United Kingdom r.mitkov@wlv.ac.uk Abstract In this paper we argue that comparative evaluation in anaphora resolution has to be performed using the same pre-processing tools and on the same set of data. The paper proposes an evaluation environment for comparing anaphora resolution algorithms which is illustrated by presenting the results of the comparative evaluation of three methods on the basis of several evaluation measures. 1 Introduction The evaluation of any NLP algorithm or system should indicate not only its efficiency or performance, but should also help us discover what a new approach brings to the current state of play in the field. To this end, a comparative evaluation with other well-known or similar approaches would be highly desirable. We have already voiced concern (Mitkov, 1998a), (Mitkov, 2000b) that the evaluation of anaphora resolution algorithms and systems is bereft of any common ground for comparison due not only to the difference of the evaluation data, but also due to the diversity of pre-processing tools employed by each anaphora resolution system. The evaluation picture would not be accurate even if we compared anaphora resolution systems on the basis of the same data since the pre-processing errors which would be carried over to the systems’ outputs might vary. As a way forward we have proposed the idea of the evaluation workbench (Mitkov, 2000b) - an open-ended architecture which allows the incorporation of different algorithms and their comparison on the basis of the same pre-processing tools and the same data. Our paper discusses a particular configuration of this new evaluation environment incorporating three approaches sharing a common ”knowledge-poor philosophy”: Kennedy and Boguraev’s (1996) parser-free algorithm, Baldwin’s (1997) CogNiac and Mitkov’s (1998b) knowledge-poor approach. 2 The evaluation workbench for anaphora resolution In order to secure a ”fair”, consistent and accurate evaluation environment, and to address the problems identified above, we have developed an evaluation workbench for anaphora resolution which allows the comparison of anaphora resolution approaches sharing common principles (e.g. similar pre-processing or resolution strategy). The workbench enables the ”plugging in” and testing of anaphora resolution algorithms on the basis of the same pre-processing tools and data. This development is a time-consuming task, given that we have to re-implement most of the algorithms, but it is expected to achieve a clearer assessment of the advantages and disadvantages of the different approaches. Developing our own evaluation environment (and even reimplementing some of the key algorithms) also alleviates the impracticalities associated with obtaining the codes of original programs. Another advantage of the evaluation workbench is that all approaches incorporated can operate either in a fully automatic mode or on human annotated corpora. We believe that this is a consistent way forward because it would not be fair to compare the success rate of an …

Content truncated to 500 words