Error Handling in the RavenClaw Dialog Management Architecture

Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing (HLT/EMNLP), pages 225–232, Vancouver, October 2005. c⃝2005 Association for Computational Linguistics Error Handling in the RavenClaw Dialog Management Framework Dan Bohus Computer Science Department Carnegie Mellon University Pittsburgh, PA, 15213 dbohus@cs.cmu.edu Alexander I. Rudnicky Computer Science Department Carnegie Mellon University Pittsburgh, PA, 15213 air@cs.cmu.edu Abstract We describe the error handling architect- ture underlying the RavenClaw dialog management framework. The architecture provides a robust basis for current and fu- ture research in error detection and recov- ery. Several objectives were pursued in its development: task-independence, ease-of- use, adaptability and scalability. We de- scribe the key aspects of architectural de- sign which confer these properties, and discuss the deployment of this architect- ture in a number of spoken dialog systems spanning several domains and interaction types. Finally, we outline current research projects supported by this architecture. 1 Introduction Over the last decade, improvements in speech rec- ognition and other component technologies have paved the way for the emergence of complex task- oriented spoken dialog systems. While traditionally the research community has focused on building information-access and command-and-control systems, recent efforts aim towards building more sophisticated language-enabled agents, such as personal assistants, interactive tutors, open-domain question answering systems, etc. At the other end of the complexity spectrum, simpler systems have already transitioned into day-to-day use and are becoming the norm in the phone-based customer- service industry. Nevertheless, a number of problems remain in need of better solutions. One of the most important limitations in today’s spoken language interfaces is their lack of robustness when faced with under- standing errors. This problem appears across all domains and interaction types, and stems primarily from the inherent unreliability of the speech recog- nition process. The recognition difficulties are further exacerbated by the conditions under which these systems typically operate: spontaneous spe- ech, large vocabularies and user populations, and large variability in input line quality. In these set- tings, average word-error-rates of 20-30% (and up to 50% for non-native speakers) are quite common. Left unchecked, speech recognition errors can lead to two types of problems in a spoken dialog system: misunderstandings and non-understand- ings. In a misunderstanding, the system obtains an incorrect semantic interpretation of the user’s turn. In the absence of robust mechanisms for assessing the reliability of the decoded inputs, the system will take the misunderstanding as fact and will act based on invalid information. In contrast, in a non- understanding the system fails to obtain an inter- pretation of the input altogether. Although no false information is incorporated in this case, the situa- tion is not much better: without an appropriate set of recovery strategies and a mechanism for diag- nosing the problem, the system’s follow-up options are limited and uninformed. In general, unless mitigated by accurate error awareness and robust recovery mechanisms, speech recognition errors exert a strong negative impact on the quality and ultimately on the success of the interactions (Sand- ers et al, 2002). Two pathways …

Content truncated to 500 words