Evaluating Discourse Processing Algorithms

EVALUATING DISCOURSE PROCESSING ALGORITHMS Marilyn A. Walker Hewlett Packard Laboratories Filton Rd., Bristol, England B$12 6QZ, U.K. & University of Pennsylvania lyn%lwalker~hplb.hpl.hp.com Abstract In order to take steps towards establishing a method- ology for evaluating Natural Language systems, we conducted a case study. We attempt to evaluate two different approaches to anaphoric processing in dis- course by comparing the accuracy and coverage of two published algorithms for finding the co-specifiers of pronouns in naturally occurring texts and dia- logues. We present the quantitative results of hand- simulating these algorithms, but this analysis natu- rally gives rise to both a qualitative evaluation and recommendations for performing such evaluations in general. We illustrate the general difficulties encoun- tered with quantitative evaluation. These are prob- lems with: (a) allowing for underlying assumptions, (b) determining how to handle underspecifications, and (c) evaluating the contribution of false positives and error chaining. 1 Introduction In the course of developing natural language inter- faces, computational linguists are often in the posi- tion of evaluating different theoretical approaches to the analysis of natural language (NL). They might want to (a) evaluate and improve on a current sys- tem, (b) add a capability to a system that it didn't previously have, (c) combine modules from different systems. Consider the goal of adding a discourse compo- nent to a system, or evaluating and improving one that is already in place. A discourse module might combine theories on, e.g., centering or local focus- ing [GJW83, Sid79], global focus [Gro77], coher- ence relations[Hob85], event" reference [Web86], in- tonational structure [PH87], system vs. user be- liefs [Po186], plan or intent recognition or production [(3o578, AP86, SIS1], control[WSSS], or complex syn- tactic structures [Pri85]. How might one evaluate the relative contributions of each of these factors or com- pare two approaches to the same problem? In order to take steps towards establishing a methodology for doing this type of comparison, we conducted a case study. We attempt to evalu- ate two different approaches to anaphoric processing in discourse by comparing the accuracy and cover- age of two published algorithms for finding the co- specifiers of pronouns in naturally occurring texts and dialogues[Hob76b, BFP87]. Thus there are two parts to this paper: we present the quantitative results of hand-simulating these algorithms (henceforth Hobbs algorithm and BFP algorithm), but this analysis nat- urally gives rise to both a qualitative evaluation and recommendations for performing such evaluations in general. We illustrate the general difficulties encoun- tered with quantitative evaluation. These are prob- lems with: (a) allowing for underlying assumptions, (b) determining how to handle underspecifications, and (c) evaluating the contribution of false positives and error chaining. Although both algorithms are part of theories of discourse that posit the interaction of the algorithm with an inference or intentional component, we will not use reasoning in tandem with the algorithm's op- eration. We have made this choice because we want to be able to analyse the performance of the algo- rithms across different domains. We focus on the linguistic …

Content truncated to 500 words