Dependencies between Student State and Speech Recognition Problems in Spoken Tutoring Dialogues

Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the ACL, pages 193–200, Sydney, July 2006. c⃝2006 Association for Computational Linguistics Dependencies between Student State and Speech Recognition Problems in Spoken Tutoring Dialogues Mihai Rotaru University of Pittsburgh Pittsburgh, USA mrotaru@cs.pitt.edu Diane J. Litman University of Pittsburgh Pittsburgh, USA litman@cs.pitt.edu Abstract Speech recognition problems are a reality in current spoken dialogue systems. In order to better understand these phenom- ena, we study dependencies between speech recognition problems and several higher level dialogue factors that define our notion of student state: frustra- tion/anger, certainty and correctness. We apply Chi Square (χ2) analysis to a cor- pus of speech-based computer tutoring dialogues to discover these dependencies both within and across turns. Significant dependencies are combined to produce interesting insights regarding speech rec- ognition problems and to propose new strategies for handling these problems. We also find that tutoring, as a new do- main for speech applications, exhibits in- teresting tradeoffs and new factors to consider for spoken dialogue design. 1 Introduction Designing a spoken dialogue system involves many non-trivial decisions. One factor that the designer has to take into account is the presence of speech recognition problems (SRP). Previous work (Walker et al., 2000) has shown that the number of SRP is negatively correlated with overall user satisfaction. Given the negative im- pact of SRP, there has been a lot of work in try- ing to understand this phenomenon and its impli- cations for building dialogue systems. Most of the previous work has focused on lower level details of SRP: identifying components responsi- ble for SRP (acoustic model, language model, search algorithm (Chase, 1997)) or prosodic characterization of SRP (Hirschberg et al., 2004). We extend previous work by analyzing the re- lationship between SRP and higher level dia- logue factors. Recent work has shown that dia- logue design can benefit from several higher level dialogue factors: dialogue acts (Frampton and Lemon, 2005; Walker et al., 2001), prag- matic plausibility (Gabsdil and Lemon, 2004). Also, it is widely believed that user emotions, as another example of higher level factor, interact with SRP but, currently, there is little hard evi- dence to support this intuition. We perform our analysis on three high level dialogue factors: frustration/anger, certainty and correctness. Frus- tration and anger have been observed as the most frequent emotional class in many dialogue sys- tems (Ang et al., 2002) and are associated with a higher word error rate (Bulyko et al., 2005). For this reason, we use the presence of emotions like frustration and anger as our first dialogue factor. Our other two factors are inspired by another contribution of our study: looking at speech- based computer tutoring dialogues instead of more commonly used information retrieval dia- logues. Implementing spoken dialogue systems in a new domain has shown that many practices do not port well to the new domain (e.g. confir- mation of long prompts (Kearns et al., 2002)). Tutoring, as a new domain for speech applica- …

Content truncated to 500 words