Evaluation of Spoken Language Systems: the ATIS Domain

Evaluation of Spoken Language Systems: the ATIS Domain P. J. Price SRI International 333 Ravenswood Ave. Menlo Park, CA 94025 Abstract Progress can be measured and encouraged via standards for comparison and evaluation. Though qualitative as- sessments can be useful in initial stages, quantifiable measures of systems under the same conditions are es- sential for comparing results and assessing claims. This paper will address the emerging standards for evaluation of spoken language systems. Introduction and Background Numbers are meaningless unless it is clear where they come from. The evaluation of any technology is greatly enhanced in usefulness if accompanied by documented standards for assessment. There has been a growing ap- preciation in the speech recognition community of the importance of standards for reporting performance. The availability of standard databases and protocols for eval- uation has been an important component in progress in the field and in the sharing of new ideas. Progress toward evaluating spoken language systems, like the technology itself, is beginning to emerge. This paper presents some background on the problem and outlines the issues and initial experiments in evaluating spoken language sys- tems in the "common" task domain, known as ATIS (Air Travel Information Service). The speech recognition community has reached agree- ment on some standards for evaluating speech recogni- tion systems, and is beginning to evolve a mechanism for revising these standards as the needs of the community change (e.g., as new systems require new kinds of data, as new system capabilities emerge, or as refinements in ex- isting methods develop). A protocol for testing speaker- dependent and speaker-independent speech recognition systems on read speech with a 1000-word vocabulary, (e.g., [6]), coordinated through the National Institute of Standards and Technology (NIST), has been operating for several years. This mechanism has inspired a healthy environment of competitive cooperation, and has led to documented major performance improvements and has increased the sharing of methodologies and of data. Evaluation of natural language (NL) understanding is more difficult than recognition because (1) the phe- nomena of interest occur less frequently (a given corpus contains more phones and words than syntactic or se- mantic phenomena), (2) semantics is far more domain dependent than phonetics or phonology, hence changing domains is more labor intensive, and (3) there is less agreement on what constitutes the "correct" analysis. However, MUCK, Message Understanding Conference, is planning the third in a series of message understand- ing evaluations for later this year (August 1990). The objective is to carry out evaluations of text interpreta- tion systems. The previous evaluation, carried out in March-June 1989, yielded quantitative measures of per- formance for eight natural language processing systems [4, 5]. The systems are evaluated on performance on a template-filling task and scored on measures of com- pleteness and precision [7]. So far, we have discussed the evaluation of automatic speech recognition (i.e., the algorithmic translation from human speech to machine readable text), and of some aspects of natural language understanding (i.e., the au- tomatic computation of a meaning and …

Content truncated to 500 words