MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 527–536 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 527 MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations Soujanya Poria†, Devamanyu HazarikaΦ, Navonil Majumder‡, Gautam Naik¶, Erik Cambria¶, Rada Mihalceaι †Information Systems Technology and Design, SUTD, Singapore ΦSchool of Computing, National University of Singapore, Singapore ‡Centro de Investigaci´on en Computaci´on, Instituto Polit´ecnico Nacional, Mexico ¶Computer Science & Engineering, Nanyang Technological University, Singapore ιComputer Science & Engineering, University of Michigan, USA sporia@sutd.edu.sg, hazarika@comp.nus.edu.sg, navo@nlp.cic.ipn.mx, gautam@sentic.net, cambria@ntu.edu.sg, mihalcea@umich.edu Abstract Emotion recognition in conversations (ERC) is a challenging task that has recently gained pop- ularity due to its potential applications. Un- til now, however, there has been no large- scale multimodal multi-party emotional con- versational database containing more than two speakers per dialogue. To address this gap, we propose the Multimodal EmotionLines Dataset (MELD), an extension and enhance- ment of EmotionLines. MELD contains about 13,000 utterances from 1,433 dialogues from the TV-series Friends. Each utterance is an- notated with emotion and sentiment labels, and encompasses audio, visual, and textual modalities. We propose several strong mul- timodal baselines and show the importance of contextual and multimodal information for emotion recognition in conversations. The full dataset is available for use at http:// affective-meld.github.io. 1 Introduction With the rapid growth of Artificial Intelligence (AI), multimodal emotion recognition has become a ma- jor research topic, primarily due to its potential applications in many challenging tasks, such as dialogue generation, user behavior understanding, multimodal interaction, and others. A conversa- tional emotion recognition system can be used to generate appropriate responses by analyzing user emotions (Zhou et al., 2017; Rashkin et al., 2018). Although significant research work has been car- ried out on multimodal emotion recognition using audio, visual, and text modalities (Zadeh et al., 2016a; Wollmer et al., 2013), significantly less work has been devoted to emotion recognition in conversations (ERC). One main reason for this is the lack of a large multimodal conversational dataset. According to Poria et al. (2019), ERC presents several challenges such as conversational context modeling, emotion shift of the interlocutors, and others, which make the task more difficult to ad- dress. Recent work proposes solutions based on multimodal memory networks (Hazarika et al., 2018). However, they are mostly limited to dyadic conversations, and thus not scalable to ERC with multiple interlocutors. This calls for a multi-party conversational data resource that can encourage research in this direction. In a conversation, the participants’ utterances generally depend on their conversational context. This is also true for their associated emotions. In other words, the context acts as a set of parameters that may influence a person to speak an utterance while expressing a certain emotion. Modeling this context can be done in different ways, e.g., by us- ing recurrent neural networks (RNNs) and mem- ory networks (Hazarika et al., 2018; Poria et al., 2017; Serban et al., 2017). Figure 1 shows an ex- ample where the speakers …

Content truncated to 500 words