M3ED: Multi-modal Multi-scene Multi-label Emotional Dialogue Database

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 5699 - 5710 May 22-27, 2022 c⃝2022 Association for Computational Linguistics M3ED: Multi-modal Multi-scene Multi-label Emotional Dialogue Database Jinming Zhao1, Tenggan Zhang1, Jingwen Hu1, Yuchen Liu1, Qin Jin1∗ Xinchao Wang3, Haizhou Li2,3 1 School of Information, Renmin University of China 2 School of Data Science, The Chinese University of Hong Kong, Shenzhen, China 3 Electrical and Computer Engineering, National University of Singapore Abstract The emotional state of a speaker can be influ- enced by many different factors in dialogues, such as dialogue scene, dialogue topic, and interlocutor stimulus. The currently available data resources to support such multimodal af- fective analysis in dialogues are however lim- ited in scale and diversity. In this work, we pro- pose a Multi-modal Multi-scene Multi-label Emotional Dialogue dataset, M3ED, which contains 990 dyadic emotional dialogues from 56 different TV series, a total of 9,082 turns and 24,449 utterances. M3ED is annotated with 7 emotion categories (happy, surprise, sad, disgust, anger, fear, and neutral) at utterance level, and encompasses acoustic, visual, and textual modalities. To the best of our knowl- edge, M3ED is the first multimodal emotional dialogue dataset in Chinese. It is valuable for cross-culture emotion analysis and recognition. We apply several state-of-the-art methods on the M3ED dataset to verify the validity and quality of the dataset. We also propose a gen- eral Multimodal Dialogue-aware Interaction framework, MDI, to model the dialogue con- text for emotion recognition, which achieves comparable performance to the state-of-the-art methods on the M3ED. The full dataset and codes are available1. 1 Introduction Emotion Recognition in Conversation (ERC) aims to automatically identify and track the emotional status of speakers during a dialogue (Poria et al., 2019b). It is a crucial component to improve natural human-computer interactions and has a wide range of applications in interaction scenar- ios, including call-center dialogue systems (Danieli et al., 2015), conversational agents (Fragopana- gos and Taylor, 2005) and mental health diagnoses (Ringeval et al., 2018), etc. Different from tradi- tional multimodal emotion recognition on isolated *Corresponding Author 1https://github.com/AIM3-RUC/RUCM3ED (Sad) !"#$%&'( Xiaoan, I am sorry (Surprise) )*+,-./0 What are you hiding from me? (Neutral) .123+,-4( I am hiding something from you (Neutral) 41%12)*+,-.0 Are you hiding something from me? (Sad) 52+6784( I didn‘t tell you in advance. (Fear, Sad) 9:.#.; Actually# I#I… (Fear, Sad) .<=>( I was divorced. (Surprise) )*04?=>0 What? You were married? (Sad) .?=>( I was married. (Fear) %=.@A<'#.<'( But I‘m divorced now, I’m divorced. (Fear) "B#4C.D#4EF( Anjie, listen to me, sit down first. (Anger) 4GHIJK You liar! Figure 1: An example of a dialogue, showing the rich emotions, inter and intra-turn emotion shifts, emotional inertia and blended emotions. utterances, multimodal ERC is a more challeng- ing problem, because there are many influencing factors that affect the speakers’ emotional state in a dialogue, including the dialogue context from multi-modalities, the scene, the topic, and even the personality of subjects, etc. (Poria et al., 2019b; Scherer, 2005; Koval et …

Content truncated to 500 words