What’s the Trouble: Automatically Identifying Problematic Dialogues in DARPA Communicator Dialogue Systems

What’s the Trouble: Automatically Identifying Problematic Dialogues in DARPA Communicator Dialogue Systems Helen Wright Hastie, Rashmi Prasad, Marilyn Walker AT& T Labs - Research 180 Park Ave, Florham Park, N.J. 07932, U.S.A. hhastie,rjprasad,walker@research.att.com Abstract Spoken dialogue systems promise effi- cient and natural access to information services from any phone. Recently, spo- ken dialogue systems for widely used ap- plications such as email, travel informa- tion, and customer care have moved from research labs into commercial use. These applications can receive millions of calls a month. This huge amount of spoken dialogue data has led to a need for fully automatic methods for selecting a subset of caller dialogues that are most likely to be useful for further system improve- ment, to be stored, transcribed and further analyzed. This paper reports results on automatically training a Problematic Di- alogue Identifier to classify problematic human-computer dialogues using a corpus of 1242 DARPA Communicator dialogues in the travel planning domain. We show that using fully automatic features we can identify classes of problematic dialogues with accuracies from 67% to 89%. 1 Introduction Spoken dialogue systems promise efficient and nat- ural access to a large variety of information services from any phone. Deployed systems and research prototypes exist for applications such as personal email and calendars, travel and restaurant informa- tion, personal banking, and customer care. Within the last few years, several spoken dialogue systems for widely used applications have moved from re- search labs into commercial use (Baggia et al., 1998; Gorin et al., 1997). These applications can receive millions of calls a month. There is a strong require- ment for automatic methods to identify and extract dialogues that provide training data for further sys- tem development. As a spoken dialogue system is developed, it is first tested as a prototype, then fielded in a limited setting, possibly running with human supervision (Gorin et al., 1997), and finally deployed. At each stage from research prototype to deployed commer- cial application, the system is constantly undergoing further development. When a system is prototyped in house or first tested in the field, human subjects are often paid to use the system and give detailed feedback on task completion and user satisfaction (Baggia et al., 1998; Walker et al., 2001). Even when a system is deployed, it often keeps evolving, either because customers want to do different things with it, or because new tasks arise out of develop- ments in the underlying application. However, real customers of a deployed system may not be willing to give detailed feedback. Thus, the widespread use of these systems has created a data management and analysis problem. System designers need to constantly track system performance, identify problems, and fix them. Sys- tem modules such as automatic speech recognition (ASR), natural language understanding (NLU) and dialogue management may rely on training data col- lected at each phase. ASR performance assessment relies on full transcription of the utterances. Dia- logue manager assessment relies on a human inter- face expert reading a full …

Content truncated to 500 words