AVAB-DBS: an Audio-Visual Affect Bursts Database for Synthesis

AVAB-DBS: Audio-Visual Affect Bursts Database for Synthesis Kevin El Haddad, H¨useyin C¸ akmak, St´ephane Dupont, Thierry Dutoit TCTS lab - University of Mons 31 Boulevard Dolez, 7000, Mons Belgium {kevin.elhaddad}{huseyin.cakmak}{stephane.dupont}{thierry.dutoit}@umons.ac.be Abstract It has been shown that adding expressivity and emotional expressions to an agent’s communication systems would improve the interaction quality between this agent and a human user. In this paper we present a multimodal database of affect bursts, which are very short non-verbal expressions with facial, vocal, and gestural components that are highly synchronized and triggered by an identifiable event. This database contains motion capture and audio data of affect bursts representing disgust, startle and surprise recorded at three different levels of arousal each. This database is to be used for synthesis purposes in order to generate affect bursts of these emotions on a continuous arousal level scale. Keywords: Affect burst, Multimodal Database, Synthesis 1. Introduction It has been mentioned in several recent studies that adding expressivity and emotional expressions to an agent’s com- munication systems would improve the interaction quality between this agent and a human user. Although emotions can be expressed in several and different ways, most of the work aiming at improving an agent’s expressions natural- ness, focus on speech (with or without the corresponding facial expressions) (Jia et al., 2011) or isolated facial ex- pressions (Moosaei et al., 2014). In this work we focus on non-verbal audio-visual expressions. Affect bursts are de- fined in (Krumhuber and Scherer, 2011; Scherer, 1994) as very short non-verbal expressions with facial, vocal, and gestural components that are highly synchronized and trig- gered by an identifiable event. Emotional expressions are culturally, socially and even in- dividually variable. This means that a particular emotion will be expressed in different ways and modalities at dif- ferent times. From this, comes the difficulty of using a naturalistic database with parametric modeling techniques for synthesis purposes. Indeed, the data recorded for a cer- tain emotional expression in a naturalistic database will be different from one another. Therefore, although being ad- equate for recognition and classification tasks (if the data quantity is enough with respect to the task) this kind of data would most probably not be suitable to create parametric models such as Hidden Markov Models (HMM) (Rabiner and Juang, 1986) or Deep Neural Networks (DNN) (Hin- ton and Osindero, 2006; Anderson, 1995) with the goal of generating descriptors. This difficulty concerns not only modeling a specific emotion expression without taking the arousal level of the emotion into account, but also doing so on continuous arousal scale. Being able to synthesize emo- tion expressions on a continuous arousal level scale, will enable more control over the avatar’s expressiveness. In order to do so, we present an Audio-Visual (AV) database collected for the purpose of synthesizing AV expressions at different levels via statistical parametric modeling, i.e. us- ing HMM in our case. Indeed previous work proved HMM- based systems to be efficient for such applications (Cakmak et al., 2015). We will first discuss our motivations …

Content truncated to 500 words