Latent Mixture of Discriminative Experts for Multimodal Prediction Modeling

Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), pages 860–868, Beijing, August 2010 Latent Mixture of Discriminative Experts for Multimodal Prediction Modeling Derya Ozkan, Kenji Sagae and Louis-Philippe Morency USC Institute for Creative Technologies {ozkan,sagae,morency}@ict.usc.edu Abstract During face-to-face conversation, people naturally integrate speech, gestures and higher level language interpretations to predict the right time to start talking or to give backchannel feedback. In this paper we introduce a new model called Latent Mixture of Discriminative Experts which addresses some of the key issues with multimodal language processing: (1) temporal synchrony/asynchrony between modalities, (2) micro dynamics and (3) in- tegration of different levels of interpreta- tion. We present an empirical evaluation on listener nonverbal feedback prediction (e.g., head nod), based on observable be- haviors of the speaker. We confirm the im- portance of combining four types of mul- timodal features: lexical, syntactic struc- ture, eye gaze, and prosody. We show that our Latent Mixture of Discriminative Experts model outperforms previous ap- proaches based on Conditional Random Fields (CRFs) and Latent-Dynamic CRFs. 1 Introduction Face-to-face communication is highly interactive. Even when only one person speaks at a time, other participants exchange information continu- ously amongst themselves and with the speaker through gestures, gaze and prosody. These differ- ent channels contain complementary information essential to interpretation and understanding of human behaviors (Oviatt, 1999). Psycholinguistic studies also suggest that gesture and speech come from a single underlying mental process, and they Pitch Words Gaze Time P(nod) Look at listener Speaker Listener Prediction Figure 1: Example of multimodal prediction model: listener nonverbal backchannel prediction based on speaker’s speech and eye gaze. As the speaker says the word her, which is the end of the clause (her is also the object of the verb bother- ing), and lowers the pitch while looking back at the listener and eventually pausing, the listener is then very likely to head nod (i.e., nonverbal backchannel). are related both temporally and semantically (Mc- Neill, 1992; Cassell and Stone, 1999; Kendon, 2004). A good example of such complementarity is how people naturally integrate speech, gestures and higher level language to predict when to give backchannel feedback. Building computational models of such a predictive process is challeng- ing since it involves micro dynamics and temporal relationship between cues from different modali- ties (Quek, 2003). Figure 1 shows an example of backchannel prediction where a listener head nod 860 is more likely. For example, a temporal sequence from the speaker where he/she reaches the end of segment (syntactic feature) with a low pitch and looks at the listener before pausing is a good op- portunity for the listener to give nonverbal feed- back (e.g., head nod). These prediction models have broad applicability, including the improve- ment of nonverbal behavior recognition, the syn- thesis of natural animations for robots and virtual humans, the training of cultural-specific nonver- bal behaviors, and the diagnoses of social disor- ders (e.g., autism spectrum disorder). In this paper we introduce a new model called Latent Mixture of …

Content truncated to 500 words