Introducing MULAI: A Multimodal Database of Laughter during Dyadic Interactions

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 4333–4342 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 4333 Introducing MULAI: A Multimodal Database of Laughter during Dyadic Interactions Michel-Pierre Jansen∗, Khiet P. Truong∗, Deniece S. Nazareth∗ †, Dirk K.J. Heylen∗ ∗University of Twente - Human Media Interaction {m.jansen-1, k.p.truong, d.k.j.heylen}@utwente.nl †University of Twente - Psychology, Health and Technology d.s.nazareth@utwente.nl Abstract Although laughter has gained considerable interest from a diversity of research areas, there still is a need for laughter specific databases. We present the Multimodal Laughter during Interaction (MULAI) database to study the expressive patterns of conversational and humour related laughter. The MULAI database contains 2 hours and 14 minutes of recorded and annotated dyadic human-human interactions and includes 601 laughs, 168 speech-laughs and 538 on- or offset respirations. This database is unique in several ways; 1) it focuses on different types of social laughter including conversational- and humour related laughter, 2) it contains annotations from participants, who understand the social context, on how humourous they perceived themselves and their interlocutor during each task, and 3) it contains data rarely captured by other laughter databases including participant personality profiles and physiological responses. We use the MULAI database to explore the link between acoustic laughter properties and annotated humour ratings over two settings. The results reveal that the duration, pitch and intensity of laughs from participants do not correlate with their own perception of how humourous they are, however the acoustics of laughter do correlate with how humourous they are being perceived by their conversational partner. Keywords: database, dyadic, interaction, audio, video, physiological, social laughter, acoustic properties, self-ratings, perceiver- ratings, humour 1. Introduction Many theories exist about the exact origin and function of laughter (Gervais and Wilson, 2005), however most theo- rists agree that laughter plays a major role in day to day social interactions between humans. Laughter can be com- plex and displayed through multiple modalities including vocal expressions, facial expressions and body movements. Some research points out that (prosodic) laughter proper- ties differ depending on the social function and situation of the laughter (Campbell, 2007; Tanaka and Campbell, 2011), whereas others (Curran et al., 2018; Owren and Ba- chorowski, 2003) argue that for most prosodic features the evidence is inconclusive and point towards social context for a better understanding of laughter. In addition, knowl- edge about how multimodal signals of expressive patterns of laughter interact with each other is still rather limited (Niewiadomski et al., 2013). Hence, to study laughter and its expressive patterns within the social context, databases with social interactions are needed that contain annotated laughter sequences. Al- though several databases containing annotated laughter are available to the research community, they are often not specifically designed for the research of functionally differ- ent laughs within a social setting. The difficulty of study- ing laughter from such databases is that laughs often are elicited outside of a social context and therefore miss the specific nuances, different functional properties, …

Content truncated to 500 words