Predicting Emotion in Spoken Dialogue from Multiple Knowledge Sources

Predicting Emotion in Spoken Dialogue from Multiple Knowledge Sources Kate Forbes-Riley University of Pittsburgh Learning Research and Development Center Pittsburgh PA, 15260, USA forbesk@pitt.edu Diane J. Litman University of Pittsburgh Department of Computer Science Learning Research and Development Center Pittsburgh PA, 15260, USA litman@cs.pitt.edu Abstract We examine the utility of multiple types of turn-level and contextual linguistic features for automatically predicting student emotions in human-human spoken tutoring dialogues. We first annotate student turns in our corpus for negative, neutral and positive emotions. We then automatically extract features represent- ing acoustic-prosodic and other linguistic in- formation from the speech signal and associ- ated transcriptions. We compare the results of machine learning experiments using different feature sets to predict the annotated emotions. Our best performing feature set contains both acoustic-prosodic and other types of linguistic features, extracted from both the current turn and a context of previous student turns, and yields a prediction accuracy of 84.75%, which is a 44% relative improvement in error reduc- tion over a baseline. Our results suggest that the intelligent tutoring spoken dialogue system we are developing can be enhanced to automat- ically predict and adapt to student emotions. 1 Introduction This paper investigates the automatic classification of student emotional states using acoustic-prosodic, non- acoustic-prosodic, and contextual information, in a cor- pus of human-human spoken tutoring dialogues. Motiva- tion for this work comes from the discrepancy between the performance of human tutors and current machine tutors. In recent years, the development of computa- tional tutorial dialogue systems has become more preva- lent (Aleven and Rose, 2003), as one method of attempt- ing to close the performance gap between human and computer tutors. It has been hypothesized that the suc- cess of such computer dialogue tutors could be further increased by modeling and adapting to student emotion; for example (Aist et al., 2002) have shown that adding human-provided emotional scaffolding to an automated reading tutor increases student persistence. We are build- ing an intelligent tutoring spoken dialogue system with the goal of using spoken and natural language processing capabilities to automatically predict and adapt to student emotions. Here we present results of an empirical study demonstrating the feasibility of modeling student emo- tion in a corresponding corpus of human-human spoken tutoring dialogues. Research in emotional speech has already shown that acoustic and prosodic features can be extracted from the speech signal and used to develop predictive models of emotion. Much of this research has used databases of speech read by actors or native speakers as training data (often with semantically neutral content) (Oudeyer, 2002; Polzin and Waibel, 1998; Liscombe et al., 2003). How- ever, such prototypical emotional speech does not neces- sarily reflect natural speech (Batliner et al., 2003), such as found in tutoring dialogues. When actors are asked to read the same sentence with different emotions, they are restricted to conveying emotion using only acoustic and prosodic features. In natural interactions, however, speakers can convey emotions using other types of fea- tures, and can also combine acoustic-prosodic …

Content truncated to 500 words