DATE: A Dialogue Act Tagging Scheme for Evaluation of Spoken Dialogue Systems

DATE: A Dialogue Act Tagging Scheme for Evaluation of Spoken Dialogue Systems Marilyn Walker and Rebecca Passonneau AT&T Shannon Labs 180 Park Ave. Florham Park, N.J. 07932 fwalker,beckyg@research.att.com ABSTRACT This paper describes a dialogue act tagging scheme developed for the purpose of providing finer-grained quantitative dialogue met- rics for comparing and evaluating DARPA COMMUNICATOR spo- ken dialogue systems. We show that these dialogue act metrics can be used to quantify the amount of effort spent in a dialogue main- taining the channel of communication or, establishing the frame for communication, as opposed to actually carrying out the travel planning task that the system is designed to support. We show that the use of these metrics results in a 7% improvement in the fit in models of user satisfaction. We suggest that dialogue act metrics can ultimately support more focused qualitative analysis of the role of various dialogue strategy parameters, e.g. initiative, across di- alogue systems, thus clarifying what development paths might be feasible for enhancing user satisfaction in future versions of these systems. 1. INTRODUCTION Recent research on dialogue is based on the assumption that di- alogue acts provide a useful way of characterizing dialogue behav- iors in human-human dialogue, and potentially in human-computer dialogue as well [16, 27, 11, 7, 1]. Several research efforts have explored the use of dialogue act tagging schemes for tasks such as improving recognition performance [27], identifying important parts of a dialogue [12], and as a constraint on nominal expres- sion generation [17]. This paper reports on the development and use of a dialogue act tagging scheme for a rather different task: the evaluation and comparison of spoken dialogue systems in the travel domain. We call this scheme DATE: Dialogue Act Tagging for Evaluation. Our research on the use of dialogue act tagging for evaluation focuses on the corpus of DARPA COMMUNICATOR dialogues col- lected in the June 2000 data collection [28]. This corpus consists of 662 dialogues from 72 users calling the nine different COMMUNI- CATOR travel planning systems. Each system implemented a log- file standard for logging system behaviors and calculating a set of core metrics. Each system utterance and each recognizer result was logged, and user utterances were transcribed and incorporated into . the logfiles. The logfile standard supported the calculation of met- rics that were hypothesized to potentially affect the user’s percep- tion of the system; these included task duration, per turn measures, response latency measures and ASR performance measures. Each dialogue was also hand labelled for task completion. The hypothesis underlying our approach is that a system’s di- alogue behaviors have a strong effect on the user’s perception of the system. Yet the core metrics that were collected via the logfile standard represent very little about dialogue behaviors. For exam- ple, the logging counts system turns and tallies their average length, but doesn’t distinguish turns that reprompt the user, or give in- structions, from those that present flight information. Furthermore, each COMMUNICATOR system had a unique dialogue strategy and …

Content truncated to 500 words