Subject-Based Evaluation Measures for Interactive Spoken Language Systems

SUBJECT-BASED EVALUATION MEASURES FOR INTERACTIVE SPOKEN LANGUAGE SYSTEMS Patti Price, 1 Lynette Hirschman, 2 Elizabeth Shriberg, 3 Elizabeth Wade 4 1SRI International, 333 Ravenswood Ave., EJ 133, Menlo Park, CA 94306. 2 MIT Laboratory for Computer Science, Cambridge, MA 02139 3University of California at Berkeley, Department of Psychology, Berkeley, CA94720 4Stanford Umversity, Department of Psychology, Stanford, CA 94305 ABSTRACT The DARPA Spoken Language effort has profited greatly from its emphasis on tasks and common evaluation metrics. Common, standardized evaluation procedures have helped the community to focus research effort, to measure progress, and to encourage com- munication among participating sites. The task and the evaluation metrics, however, must be consistent with the goals of the Spoken Language program, namely interactive problem solving. Our eval- uation methods have evolved with the technology, moving from evaluation of read speech from a fixed corpus through evaluation of isolated canned sentences to evaluation of spontaneous speech in context in a canned corpus. A key component missed in current evaluations is the role of subject interaction with the system. Because of the great variability across subjects, however, it is nec- essary to use either a large number of subjects or a within-subject design. This paper proposes a within-subject design comparing the results of a software-sharing exercise carried out jointly by M1T and SRI. 1. INTRODUCTION The use of a common task and a common set of evaluation metrics has been a comerstone of DARPA-funded research in speech and spoken language systems. This approach allows researchers to evaluate and compare alternative techniques and to learn from each other's successes and failures. The choice of metrics for evaluation is a crucial component of the research program, since there will be strong pressure to make improvements with respect to the metric used. Therefore, we must select metrics carefully if they are to be relevant both to our research goals and to transition of the technology from the laboratory into appli- cations. The program goal of the Spoken Language Systems (SLS) effort is to support human-computer interactive problem solving. The DARPA SLS community has made significant progress toward this goal, and the development of appropri- ate evaluation metrics has played a key role in this effort. We have moved from evaluation of closed vocabulary, read speech (resource management) for speech recognition eval- uation to open vocabulary for spontaneous speech (ATIS). In June 1990, the first SLS dry run evaluated only tran- scribed spoken input for sentences that could be interpreted independent of context. At the DARPA workshop in Febru- ary 1991, researchers reported on speech recognition, spo- ken language understanding, and natural language understanding results for context-independent sentences and also for pairs of context-setting + context-dependent sentences. At the present workshop, we witness another major step: we are evaluating systems on speech, spoken language and natural language for all evaluable utterances within entire dialogues, requiring that systems handle each sentence in its dialogue context, with no externally sup- plied context classification information. 2. EVALUATION METHODOLOGY: WHERE ARE WE? The current …

Content truncated to 500 words