Quantitative and Qualitative Evaluation of Darpa Communicator Spoken Dialogue Systems

Quantitative and Qualitative Evaluation of Darpa Communicator Spoken Dialogue Systems Marilyn A. Walker AT&T Labs – Research 180 Park Ave, E103 Florham Park, NJ. 07932 walker@research.att.com Rebecca Passonneau AT&T Labs –Research 180 Park Ave, D191 Florham Park, NJ. 07932 becky@research.att.com Julie E. Boland Institute of Cognitive Science University of Louisiana at Lafayette Lafayette, LA 70504 boland@louisiana.edu Abstract This paper describes the application of the PARADISE evaluation framework to the corpus of 662 human-computer dialogues collected in the June 2000 Darpa Communicator data collection. We describe results based on the stan- dard logfile metrics as well as results based on additional qualitative metrics derived using the DATE dialogue act tagging scheme. We show that per- formance models derived via using the standard metrics can account for 37% of the variance in user satisfaction, and that the addition of DATE metrics im- proved the models by an absolute 5%. 1 Introduction The objective of the DARPA COMMUNICATOR program is to support research on multi-modal speech-enabled dialogue systems with advanced conversational capabilities. In order to make this a reality, it is important to understand the con- tribution of various techniques to users’ willing- ness and ability to use a spoken dialogue system. In June of 2000, we conducted an exploratory data collection experiment with nine participating communicator systems. All systems supported travel planning and utilized some form of mixed- initiative interaction. However the systems var- ied in several critical dimensions: (1) They tar- geted different back-end databases for travel in- formation; (2) System modules such as ASR, NLU, TTS and dialogue management were typ- ically different across systems. The Evaluation Committee chaired by Walker (Walker, 2000), with representatives from the nine COMMUNICATOR sites and from NIST, de- veloped the experimental design. A logfile stan- dard was developed by MITRE along with a set of tools for processing the logfiles (Aberdeen, 2000); the standard and tools were used by all sites to collect a set of core metrics for making cross system comparisons. The core metrics were developed during a workshop of the Evaluation Committee and included all metrics that anyone in the committee suggested, that could be imple- mented consistently across systems. NIST’s con- tribution was to recruit the human subjects and to implement the experimental design specified by the Evaluation Committee. The experiment was designed to make it possi- ble to apply the PARADISE evaluation framework (Walker et al., 2000), which integrates and unifies previous approaches to evaluation (Price et al., 1992; Hirschman, 2000). The framework posits that user satisfaction is the overall objective to be maximized and that task success and various in- teraction costs can be used as predictors of user satisfaction. Our results from applying PARADISE include that user satisfaction differed consider- ably across the nine systems. Subsequent model- ing of user satisfaction gave us some insight into why each system was more or less satisfactory; four variables accounted for 37% of the variance in user-satisfaction: task completion, task dura- tion, recognition accuracy, and mean system turn duration. However, …

Content truncated to 500 words