PARADISE: A Framework for Evaluating Spoken Dialogue Agents

PARADISE: A Framework for Evaluating Spoken Dialogue Agents Marilyn A. Walker, Diane J. Litman, Candace A. Kamm and Alicia Abella AT&T Labs--Research 180 Park Avenue Florham Park, NJ 07932-0971 USA walker, diane,cak,abella@research.att.com Abstract This paper presents PARADISE (PARAdigm for Dialogue System Evaluation), a general framework for evaluating spoken dialogue agents. The framework decouples task require- ments from an agent's dialogue behaviors, sup- ports comparisons among dialogue strategies, enables the calculation of performance over subdialogues and whole dialogues, specifies the relative contribution of various factors to performance, and makes it possible to compare agents performing different tasks by normaliz- ing for task complexity. 1 Introduction Recent advances in dialogue modeling, speech recogni- tion, and natural language processing have made it possi- ble to build spoken dialogue agents for a wide variety of applications, n Potential benefits of such agents include remote or hands-free access, ease of use, naturalness, and greater efficiency of interaction. However, a critical obstacle to progress in this area is the lack of a general framework for evaluating and comparing the performance of different dialogue agents. One widely used approach to evaluation is based on the notion of a reference answer (Hirschman et al., 1990). An agent's responses to a query are compared with a prede- fined key of minimum and maximum reference answers; performance is the proportion of responses that match the key. This approach has many widely acknowledged lim- itations (Hirschman and Pao, 1993; Danieli et al., 1992; Bates and Ayuso, 1993), e.g., although there may be many potential dialogue strategies for carrying out a task, the key is tied to one particular dialogue strategy. In contrast, agents using different dialogue strategies can be compared with measures such as inappropri- ate utterance ratio, turn correction ratio, concept accu- racy, implicit recovery and transaction success (Danieli LWe use the term agent to emphasize the fact that we are evaluating a speaking entity that may have a personality. Read- ers who wish to may substitute the word "system" wherever "agent" is used. and Gerbino, 1995; Hirschman and Pao, 1993; Po- lifroni et al., 1992; Simpson and Fraser, 1993; Shriberg, Wade, and Price, 1992). Consider a comparison of two train timetable information agents (Danieli and Gerbino, 1995), where Agent A in Dialogue I uses an explicit con- firmation strategy, while Agent B in Dialogue 2 uses an implicit confirmation strategy: (1) User: I want to go from Torino to Milano. Agent A: Do you want to go from Trento to Milano? Yes or No? User: No. (2) User: I want to travel from Torino to Milano. Agent B: At which time do you want to leave from Merano to Milano? User: No, I want to leave from Torino in the evening. Danieli and Gerbino found that Agent A had a higher transaction success rate and produced less inappropriate and repair utterances than Agent B, and thus concluded that Agent A was more robust than Agent B. However, one limitation of both this approach and the reference answer approach is …

Content truncated to 500 words