Conversational Memory Network for Emotion Recognition in Dyadic Dialogue Videos

Proceedings of NAACL-HLT 2018, pages 2122–2132 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics Conversational Memory Network for Emotion Recognition in Dyadic Dialogue Videos Devamanyu Hazarika School of Computing, National University of Singapore hazarika@comp.nus.edu.sg Soujanya Poria Artificial Intelligence Initiative, A*STAR, Singapore sporia@ihpc.a-star.edu.sg Amir Zadeh Language Technologies Institute, CMU, USA abagherz@cs.cmu.edu Erik Cambria School of Computer Science and Engineering, NTU, Singapore cambria@ntu.edu.sg Louis-Philippe Morency Language Technologies Institute, CMU, USA morency@cs.cmu.edu Roger Zimmermann School of Computing, National University of Singapore rogerz@comp.nus.edu.sg Abstract Emotion recognition in conversations is cru- cial for the development of empathetic ma- chines. Present methods mostly ignore the role of inter-speaker dependency relations while classifying emotions in conversations. In this paper, we address recognizing utterance-level emotions in dyadic conversational videos. We propose a deep neural framework, termed con- versational memory network, which leverages contextual information from the conversation history. The framework takes a multimodal approach comprising audio, visual and textual features with gated recurrent units to model past utterances of each speaker into memo- ries. Such memories are then merged using attention-based hops to capture inter-speaker dependencies. Experiments show an accuracy improvement of 3−4% over the state of the art. 1 Introduction Development of machines with emotional intelli- gence has been a long-standing goal of AI. With the increasing infusion of interactive systems in our lives, the need for empathetic machines with emotional understanding is paramount. Previous research in affective computing has looked at di- alogues as an essential basis to learn emotional dynamics (Sidnell and Stivers, 2012; Poria et al., 2017a; Zhou et al., 2017). Since the advent of Web 2.0, dialogue videos have proliferated across the internet through plat- forms like movies, webinars, and video chats. Emo- tion detection from such resources can benefit numerous fields like counseling (De Choudhury et al., 2013), public opinion mining (Cambria et al., 2017), financial forecasting (Xing et al., 2018), and intelligent systems such as smart homes and chat- bots (Young et al., 2018). In this paper, we analyze emotion detection in videos of dyadic conversations. A dyadic conver- sation is a form of a dialogue between two enti- ties. We propose a conversational memory net- work (CMN), which uses a multimodal approach for emotion detection in utterances (a unit of speech bound by breathes or pauses) of such conversa- tional videos. Emotional dynamics in a conversation is known to be driven by two prime factors: self and inter- speaker emotional influence (Morris and Keltner, 2000; Liu and Maitlis, 2014). Self-influence re- lates to the concept of emotional inertia, i.e., the degree to which a person’s feelings carry over from one moment to another (Koval and Kuppens, 2012). Inter-speaker emotional influence is another trait where the other person acts as an influencer in the speaker’s emotional state. Conversely, speak- ers also tend to mirror emotions of their counter- parts (Navarretta et al., 2016). Figure 1 provides an example from the dataset showing the presence of these two traits in a dialogue. Existing works in the literature …

Content truncated to 500 words