MEISD: A Multimodal Multi-Label Emotion, Intensity and Sentiment Dialogue Dataset for Emotion Recognition and Sentiment Analysis in Conversations

Proceedings of the 28th International Conference on Computational Linguistics, pages 4441–4453 Barcelona, Spain (Online), December 8-13, 2020 4441 MEISD: A Multimodal Multi-Label Emotion, Intensity and Sentiment Dialogue Dataset for Emotion Recognition and Sentiment Analysis in Conversations Mauajama Firdaus∗, Hardik Chauhan, Asif Ekbal and Pushpak Bhattacharyya Department of Computer Science and Engineering Indian Institute of Technology Patna, India (mauajama.pcs16,hardik,asif,pb)@iitp.ac.in Abstract Emotion and sentiment classification in dialogues is a challenging task that has gained popularity in recent times. Humans tend to have multiple emotions with varying intensities while expressing their thoughts and feelings. Emotions in an utterance of dialogue can either be independent or dependent on the previous utterances, making the task complex and interesting. Multi-label emotion detection in conversations is a significant task that provides the ability to the system to understand the various emotions of the users interacting. On the other hand, sentiment analysis in dialogue or conversation helps in understanding the perspective of the user with respect to the ongoing conversation. Besides text, additional information in the form of audio and video assists in identifying the correct emotions with the appropriate intensity and sentiments in an utterance of a dialogue. Lately, quite a few datasets have been made available for emotion and sentiment classification in dialogues. Still, these datasets are imbalanced in representing different emotions and consist of only a single emotion. Hence, we present at first a large-scale balanced Multimodal Multi-label Emotion, Intensity, and Sentiment Dialogue dataset (MEISD) collected from different TV series that has textual, audio, and visual features, and then establish a baseline setup for further research. 1 Introduction With the advancements in Artificial Intelligence (AI), the gap between Natural Language Processing (NLP) and Computer Vision (CV) has been bridged by extensive research in multi-modal information analysis. The ability to use different modalities such as text, audio and video for different tasks, such as emotion classification (Tripathi and Beigi, 2018; Hazarika et al., 2018a), sentiment analysis (Poria et al., 2017), dialogue generation (Yoshino et al., 2019; Das et al., 2017) have helped in building robust systems. The potential to understand correct emotion and sentiment in a conversation is crucial for developing strong human-machine interaction systems. Dialogue systems are of two types i.e., goal-oriented systems (Asri et al., 2017) or open chit-chat systems (Serban et al., 2017). In both these systems, understanding the user’s emotions is crucial to maximizing the user experience and satisfaction. Nowadays, there is a huge demand for developing social agents capable of having real conversations with humans. With the rapid growth in technology, personal assistants in smartphones such as Amazon’s Alexa, Apple’s Siri, and Google’s Home have become human companions. Hence, these applications need to understand the correct emotional state of the user to increase user contentment leading to user retention. Emotions and sentiments are subjective qualities and are understood to share overlapping features; hence are frequently used interchangeably. This is mainly because both sentiment and emotion refer to experiences resulting from the combination of biological, cognitive, and social influences. Though both are …

Content truncated to 500 words