Predicting User Reactions to System Error

Predicting User Reactions to System Error Diane Litman and Julia Hirschberg AT&T Labs–Research Florham Park, NJ, 07932 USA � diane/julia � @research.att.com Marc Swerts IPO, Eindhoven, The Netherlands, and CNTS, Antwerp, Belgium m.g.j.swerts@tue.nl Abstract This paper focuses on the analysis and prediction of so-called aware sites, defined as turns where a user of a spoken dialogue system first becomes aware that the system has made a speech recognition error. We describe statistical comparisons of features of these aware sites in a train timetable spoken dialogue corpus, which re- veal significant prosodic differences between such turns, compared with turns that ‘correct’ speech recogni- tion errors as well as with ‘normal’ turns that are neither aware sites nor corrections. We then present machine learning results in which we show how prosodic features in combination with other automatically available features can predict whether or not a user turn was a normal turn, a correction, and/or an aware site. 1 Introduction This paper describes new results in our continu- ing investigation of prosodic information as a po- tential resource for error recovery in interactions between a user and a spoken dialogue system. In human-human interaction, dialogue partners ap- ply sophisticated strategies to detect and correct communication failures so that errors of recog- nition and understanding rarely lead to a com- plete breakdown of the interaction (Clark and Wilkes-Gibbs, 1986). In particular, various stud- ies have shown that prosody is an important cue in avoiding such breakdown, e.g. (Shimojima et al., 1999). Human-machine interactions between a user and a spoken dialogue system (SDS) ex- hibit more frequent communication breakdowns, due mainly to errors in the Automatic Speech Re- cognition (ASR) component of these systems. In such interactions, however, there is also evidence showing prosodic information may be used as a resource for error recovery. In previous work, we identified new procedures to detect recogni- tion errors. In particular, we found that pros- odic features, in combination with other inform- ation already available to the recognizer, can dis- tinguish user turns that are misrecognized by the system far better than traditional methods used in ASR rejection (Litman et al., 2000; Hirschberg et al., 2000). We also found that user corrections of system misrecognitions exhibit certain typical prosodic features, which can be used to identify such turns (Swerts et al., 2000; Hirschberg et al., 2001). These findings are consistent with previ- ous research showing that corrections tend to be hyperarticulated — higher, louder, longer ...than other turns (Wade et al., 1992; Oviatt et al., 1996; Levow, 1998; Bell and Gustafson, 1999). In the current study, we focus on another turn category that is potentially useful in error hand- ling. In particular, we examine what we term aware sites — turns where a user, while interact- ing with a machine, first becomes aware that the system has misrecognized a previous user turn. Note that such aware sites may or may not also be corrections (another type of post-misrecognition turn), since a user may not immediately provide correcting information. …

Content truncated to 500 words