COGMEN: COntextualized GNN based Multimodal Emotion recognitioN

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4148 - 4164 July 10-15, 2022 ©2022 Association for Computational Linguistics COGMEN: COntextualized GNN based Multimodal Emotion recognitioN Abhinav Joshi Ashwani Bhat Ayush Jain Atin Vikram Singh Ashutosh Modi Indian Institute of Technology Kanpur (IIT-K) {ajoshi, ashutoshm}@cse.iitk.ac.in {ashubhat44}@gmail.com {aayushj, atinvs}@iitk.ac.in Abstract Emotions are an inherent part of human inter- actions, and consequently, it is imperative to develop AI systems that understand and rec- ognize human emotions. During a conversa- tion involving various people, a person’s emo- tions are influenced by the other speaker’s ut- terances and their own emotional state over the utterances. In this paper, we propose COntex- tualized Graph Neural Network based Multi- modal Emotion recognitioN (COGMEN) sys- tem that leverages local information (i.e., in- ter/intra dependency between speakers) and global information (context). The proposed model uses Graph Neural Network (GNN) based architecture to model the complex de- pendencies (local and global information) in a conversation. Our model gives state-of-the- art (SOTA) results on IEMOCAP and MOSEI datasets, and detailed ablation experiments show the importance of modeling information at both levels. 1 Introduction Emotions are intrinsic to humans and guide their be- havior and are indicative of the underlying thought process (Minsky, 2007). Consequently, understand- ing and recognizing emotions is vital for develop- ing AI technologies (e.g., personal digital assis- tants) that interact directly with humans. During a conversation between a number of people, there is a constant ebb and flow of emotions experienced and expressed by each person. The task of multi- modal emotion recognition addresses the problem of monitoring the emotions expressed (via various modalities, e.g., video (face), audio (speech)) by individuals in different settings such as conversa- tions. Emotions are physiological, behavioral, and communicative reactions to cognitively processed stimuli (Planalp et al., 2018). Emotions are of- ten a result of internal physiological changes, and It's a relief too. I mean I was thinking I was going to have to pack up all of my stuff but yeah. Hooray for U.S.C. Yeah. Big time. Mmhmm. Mmhmm. Yay. So what major? What are you doing? Yeah. I'm not sure. I know. We should throw a party. A rapper party ho. Yeah. Okay. I am just- Yeah. I'm sticking around, I'm just doing my thing. I'm living here. I'm so glad you're going to stay, I'm so glad you're going to be here. Yeah. Woo. Umm...P.h.D. [LAUGHTER] That's good. Don't you have to like teach a class, too, when you do that? Speaker-1 Speaker-2 Conversation Instance taken from IEMOCAP Dataset Is this a Masters or a P.h.D.? Or can you...? Nice Excited Excited Excited Excited Excited Excited Happy Excited Neutral Neutral Figure 1: An example conversation between two speak- ers, with corresponding emotions evoked for each utter- ance. these physiological reactions may not be notice- able by others and are therefore intra-personal. For example, in a conversational setting, an emotion may be a communicative reaction that has its …

Content truncated to 500 words