MuSE: a Multimodal Dataset of Stressed Emotion

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 1499–1510 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 1499 MuSE: a Multimodal Dataset of Stressed Emotion Mimansa Jaiswal1, Cristian-Paul Bara1, Yuanhang Luo1, Mihai Burzo2 Rada Mihalcea1, Emily Mower Provost1 1University of Michigan - Ann Arbor 2University of Michigan - Flint {mimansa, cpbara, royluo, mburzo, mihalcea, emilykmp}@umich.edu Abstract Endowing automated agents with the ability to provide support, entertainment and interaction with human beings requires sensing of the users’ affective state. These affective states are impacted by a combination of emotion inducers, current psychological state, and various contextual factors. Although emotion classification in both singular and dyadic settings is an established area, the effects of these additional factors on the production and perception of emotion is understudied. This paper presents a dataset, Multimodal Stressed Emotion (MuSE), to study the multimodal interplay between the presence of stress and expressions of affect. We describe the data collection protocol, the possible areas of use, and the annotations for the emotional content of the recordings. The paper also presents several baselines to measure the performance of multimodal features for emotion and stress classification. Keywords: multimodal emotion, stressed emotion, natural language, spontaneus speech 1. Introduction Virtual agents have become more integrated into our daily lives than ever before (Lucas et al., 2014). For ex- ample, Woebot is a chatbot developed to provide cognitive behavioral therapy to a user (Fitzpatrick et al., 2017). For this chatbot agent to be effective, it needs to respond dif- ferently when the user is stressed and upset versus when the user is calm and upset, which is a common strategy in counselor training (Thompson et al., 2013). While virtual agents have made successful strides in understanding the task-based intent of the user, social human-computer inter- action can still benefit from further research (Clark et al., 2019). Successful integration of virtual agents into real- life social interaction requires machines to be emotionally intelligent (Bertero et al., 2016; Yuan, 2015). But humans are complex in nature, and emotion is not expressed in isolation (Griffiths, 2003). Instead, it is af- fected by various external factors. These external factors lead to interleaved user states, which are a culmination of situational behavior, experienced emotions, psychological or physiological state, and personality traits. One of the ex- ternal factors that affects psychological state is stress. Stress can affect everyday behavior and emotion, and in severe states, is associated with delusions, depression and anxiety due to impact on emotion regulation mechanisms (Kingston and Schuurmans-Stekhoven, 2016; Schlotz et al., 2011; Tull et al., 2007; Wang and Saudino, 2011). Virtual agents can respond in accordance to users’ emotions only if the ma- chine learning systems can recognize these complex user states and correctly perceive users’ emotional intent. We introduce a dataset designed to elicit spontaneous emotional responses in the presence or absence of stress to observe and sample complex user states. There has been a rich history of visual (You et al., 2016; …

Content truncated to 500 words