Automatic Detection of Poor Speech Recognition at the Dialogue Level

Automatic Detection of Poor Speech Recognition at the Dialogue Level Diane J. Litman, Marilyn A. Walker and Michael S. Kearns AT&T Labs Research 180 Park Ave, Bldg 103 Florham Park, N.J. 07932 {diane, walker, mkearns}@research, att. com Abstract The dialogue strategies used by a spoken dialogue system strongly influence performance and user sat- isfaction. An ideal system would not use a single fixed strategy, but would adapt to the circumstances at hand. To do so, a system must be able to identify dialogue properties that suggest adaptation. This paper focuses on identifying situations where the speech recognizer is performing poorly. We adopt a machine learning approach to learn rules from a dialogue corpus for identifying these situations. Our results show a significant improvement over the baseline and illustrate that both lower-level acoustic features and higher-level dialogue features can af- fect the performance of the learning algorithm. 1 Introduction Builders of spoken dialogue systems face a number of fundamental design choices that strongly influ- ence both performance and user satisfaction. Ex- amples include choices between user, system, or mixed initiative, and between explicit and implicit confirmation of user commands. An ideal system wouldn't make such choices a priori, but rather would adapt to the circumstances at hand. For in- stance, a system detecting that a user is repeatedly uncertain about what to say might move from user to system initiative, and a system detecting that speech recognition performance is poor might switch to a dialogUe strategy with more explicit prompting, an explicit confirmation mode, or keyboard input mode. Any of these adaptations might have been appropriate in dialogue D1 from the Annie sys- tem (Kamm et al., 1998), shown in Figure 1. In order to improve performance through such adaptation, a system must first be able to identify, in real time, salient properties of an ongoing dialogue that call for some useful change in system strategy. In other words, adaptive systems should try to auto- matically identify actionable properties of ongoing dialogues. Previous work has shown that speech recognition performance is an important predictor of user satis- faction, and that changes in dialogue behavior im- pact speech recognition performance (Walker et al., 1998b; Litman et al., 1998; Kamm et al., 1998). Therefore, in this work, we focus on the task of au- tomatically detecting poor speech recognition per- formance in several spoken dialogue systems devel- oped at AT&T Labs. Rather than hand-crafting rules that classify speech recognition performance in an ongoing dialogue, we take a machine learning ap- proach. We begin with a collection of system logs from actual dialogues that were labeled by humans as having had "good" or "bad" speech recognition (the training set). We then apply standard machine learning algorithms to this training set in the hope of discovering, in a principled manner, classifiers that can automatically detect poor speech recogni- tion during novel dialogues. In order to train such classifiers, we must provide them with a number of "features" of dialogues de- rived from the system logs …

Content truncated to 500 words