Intrinsic and Extrinsic Evaluation of an Automatic User Disengagement Detector for an Uncertainty-Adaptive Spoken Dialogue System

2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 91–102, Montr´eal, Canada, June 3-8, 2012. c⃝2012 Association for Computational Linguistics Intrinsic and Extrinsic Evaluation of an Automatic User Disengagement Detector for an Uncertainty-Adaptive Spoken Dialogue System Kate Forbes-Riley and Diane Litman and Heather Friedberg and Joanna Drummond∗ University of Pittsburgh Pittsburgh, PA 15260, USA forbesk@pitt.edu, litman@pitt.edu, haf13@pitt.edu Abstract We present a model for detecting user dis- engagement during spoken dialogue interac- tions. Intrinsic evaluation of our model (i.e., with respect to a gold standard) yields results on par with prior work. However, since our goal is immediate implementation in a sys- tem that already detects and adapts to user un- certainty, we go further than prior work and present an extrinsic evaluation of our model (i.e., with respect to the real-world task). Cor- relation analyses show crucially that our au- tomatic disengagement labels correlate with system performance in the same way as the gold standard (manual) labels, while regres- sion analyses show that detecting user disen- gagement adds value over and above detecting only user uncertainty when modeling perfor- mance. Our results suggest that automatically detecting and adapting to user disengagement has the potential to significantly improve per- formance even in the presence of noise, when compared with only adapting to one affective state or ignoring affect entirely. 1 Introduction Spoken dialogue systems that can detect and adapt to user affect1 are fast becoming reality (Schuller et al., 2009b; Batliner et al., 2008; Prendinger and Ishizuka, 2005; Vidrascu and Devillers, 2005; Lee ∗Now at Univ. Toronto: jdrummond@cs.toronto.edu 1We use affect for emotions and attitudes that affect how users communicate. Other speech researchers also combine concepts of emotion, arousal, and attitudes where emotion is not full-blown (Cowie and Cornelius, 2003). and Narayanan, 2005; Shafran et al., 2003). The benefits are clear: affect-adaptive systems have been shown to increase task success (Forbes-Riley and Litman, 2011a; D’Mello et al., 2010; Wang et al., 2008) or improve other system performance met- rics such as user satisfaction (Liu and Picard, 2005; Klein et al., 2002). However, to date most affec- tive systems researchers have focused either only on affect detection, or only on detecting and adapting to a single affective state. The next step is thus to develop and evaluate spoken dialogue systems that detect and respond to multiple affective states. We previously showed that detecting and re- sponding to user uncertainty during spoken dialogue computer tutoring significantly improves task suc- cess (Forbes-Riley and Litman, 2011a). We are now taking the next step: incorporating automatic detection and adaptation to user disengagement as well, with the goal of further improving task suc- cess. We targeted user uncertainty and disengage- ment because manual annotation showed them to be the two most common user affective states in our system and both are negatively correlated with task success (Litman and Forbes-Riley, 2009; Forbes- Riley and Litman, 2011b). Thus, we hypothesize that providing appropriate responses to these states would reduce their frequency, …

Content truncated to 500 words