Construction of English-French Multimodal Affective Conversational Corpus from TV Dramas

Construction of English-French Multimodal Affective Conversational Corpus from TV Dramas Sashi Novitasari1,2, Quoc Truong Do1, Sakriani Sakti1, Dessi Lestari2, and Satoshi Nakamura1 1Graduate School of Information Science, Nara Institute of Science and Technology 8916-5 Takayama-cho, Ikoma-shi, Nara, Japan {do.truong.dj3,ssakti,s-nakamura}@is.naist.jp 2Department of Informatics, Bandung Institute of Technology, Indonesia Jl. Ganesha No.10, Bandung, Jawa Barat, Indonesia {sashinovitasari11,dessipuji}@gmail.com Abstract Recently, there has been an increase of interest in constructing corpora containing social-affective interactions. But the availability of multimodal, multilingual, and emotionally rich corpora remains limited. The tasks of recording and transcribing actual human-to-human affective conversations are also tedious and time-consuming. This paper describes construction of a multimodal affective conversational corpus based on TV dramas. The data contain parallel English-French languages in lexical, acoustic, and facial features. In addition, we annotated the part of the English data with speaker and emotion information. Our corpus can be utilized to develop and assess such tasks as speaker and emotion recognition, affective speech recognition and synthesis, linguistic, and paralinguistic speech-to-speech translation as well as a multimodal dialog system. Keywords: Corpus construction, multimodal parallel data, affective conversation, television dramas 1. Introduction Researchers have been working on spoken language pro- cessing for decades. Such technologies as speech recogni- tion, speech synthesis, speech translation, and spoken di- alog systems have been developed and progressed from a simple machine that responds to a small set of sounds to a more sophisticated artificial agent that can handle con- versational speech. Unfortunately, most of these current technologies remain limited to recognizing what was said without addressing how it was said. For example, in con- ventional speech-to-speech translation, the verbal content of speech is translated, but its non-verbal content or par- alinguistic information is ignored. On the other hand, based on text, speech, and video, re- search on emotion recognition is gaining considerable trac- tion in the fields of human-machine communication and multimedia retrieval (Schuller et al., 2009). Numerous of- ficial emotion recognition challenges (Schuller et al., 2009; Schuller et al., 2010; Schuller et al., 2011) have been held that improved the features and the classifiers that capture the traits of spoken emotions. Furthermore, different ap- proaches, which involve the characteristics of sounds and prosody based on speaking styles and the expressions of emotional speech classification in anime films, have also been proposed (Hara and Itou, 2010). However, these stud- ies only focused on non-verbal information for recogniz- ing/classifying types of emotions without addressing verbal content. Developing an artificial agent that mimics human interac- tion requires a speech-oriented interface that can handle both the verbal and non-verbal content often found in con- versations. Unfortunately, much less work has examined the technologies that consider both matters, because per- forming such emotion-affected spoken language processing tasks is not trivial. Previous studies (Williams and Stevens, 1972; Picard, 1997; Murray and Arnott, 1993) reported that emotion largely changes acoustic realization, includ- ing pitch range, speech rate, voice quality, etc. Several approaches in emotional speech recognition (Mukaihara et al., 2017) and expressive speech synthesis (Tachibana et al., 2005) have …

Content truncated to 500 words