Data-Driven Generation of Emphatic Facial Displays

Data-driven Generation of Emphatic Facial Displays Mary Ellen Foster Department of Informatics, Technical University of Munich Boltzmannstraße 3, 85748 Garching, Germany foster@in.tum.de Jon Oberlander Institute for Communicating and Collaborative Systems School of Informatics, University of Edinburgh 2 Buccleuch Place, Edinburgh EH8 9LW, United Kingdom jon@inf.ed.ac.uk Abstract We describe an implementation of data- driven selection of emphatic facial dis- plays for an embodied conversational agent in a dialogue system. A corpus of sentences in the domain of the target dia- logue system was recorded, and the facial displays used by the speaker were anno- tated. The data from those recordings was used in a range of models for generating facial displays, each model making use of a different amount of context or choosing displays differently within a context. The models were evaluated in two ways: by cross-validation against the corpus, and by asking users to rate the output. The predic- tions of the cross-validation study differed from the actual user ratings. While the cross-validation gave the highest scores to models making a majority choice within a context, the user study showed a signifi- cant preference for models that produced more variation. This preference was espe- cially strong among the female subjects. 1 Introduction It has long been documented that there are char- acteristic facial displays that accompany the em- phasised parts of spoken utterances. For example, Ekman (1979) says that eyebrow raises “appear to coincide with primary vocal stress, or more sim- ply with a word that is spoken more loudly.” Cor- relations have also been found between prosodic features and events such as head nodding and the amplitude of mouth movements. When Krah- mer and Swerts (2004) performed an empirical, cross-linguistic evaluation of the influence of brow movements on the perception of prosodic stress, they found that subjects preferred eyebrow move- ments to be correlated with the most prominent word in an utterance and that eyebrow movements boosted the perceived prominence of the word they were associated with. While many facial displays have been shown to co-occur with prosodic accents, the converse is not true: in normal embodied speech, many pitch accents and other prosodic events are unac- companied by any facial display, and when dis- plays are used, the selection varies widely. Cas- sell and Th´orisson (1999) demonstrated that “en- velope” facial displays related to the process of conversation have a greater impact on successful interaction with an embodied conversational agent than do emotional displays. However, no descrip- tion of face motion is sufficiently detailed that it can be used as the basis for selecting emphatic fa- cial displays for an agent. This is therefore a task for which data-driven techniques are beneficial. In this paper, we address the task of selecting emphatic facial displays for the talking head in the COMIC1 multimodal dialogue system. In the basic COMIC process for generating multimodal output (Foster et al., 2005), facial displays are se- lected using simple rules based only on the pitch accents specified by the text generation system. In …

Content truncated to 500 words