MMGCN: Multimodal Fusion via Deep Graph Convolution Network for Emotion Recognition in Conversation

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 5666–5675 August 1–6, 2021. ©2021 Association for Computational Linguistics 5666 MMGCN: Multimodal Fusion via Deep Graph Convolution Network for Emotion Recognition in Conversation Jingwen Hu, Yuchen Liu, Jinming Zhao, Qin Jin∗ School of Information, Renmin University of China {hujingwen benja, Liuyuchen Alfred, zhaojinming, qjin}@ruc.edu.cn Abstract Emotion recognition in conversation (ERC) is a crucial component in affective dialogue systems, which helps the system understand users’ emotions and generate empathetic re- sponses. However, most works focus on mod- eling speaker and contextual information pri- marily on the textual modality or simply lever- aging multimodal information through fea- ture concatenation. In order to explore a more effective way of utilizing both multi- modal and long-distance contextual informa- tion, we propose a new model based on mul- timodal fused graph convolutional network, MMGCN, in this work. MMGCN can not only make use of multimodal dependencies effec- tively, but also leverage speaker information to model inter-speaker and intra-speaker de- pendency. We evaluate our proposed model on two public benchmark datasets, IEMOCAP and MELD, and the results prove the effec- tiveness of MMGCN, which outperforms other SOTA methods by a significant margin under the multimodal conversation setting. 1 Introduction Emotion is an important part of human daily com- munication. Emotion Recognition in Conversation (ERC) aims to automatically identify and track the emotional status of speakers during a dialogue. It has attracted increasing attention from researchers in the field of natural language processing and mul- timodal processing. ERC has a wide range of po- tential applications such as assisting conversation analysis for legal trials and e-health services etc. It is also a key component for building natural human- computer interactions that can produce emotional responses in a dialogue. The fast growing availability of conversational data on social media is one of the factors that boost ∗Corresponding Author Oh, yeah? Have you gotten letters yet? Um, where? hang out? U.S.C.. Oh my Gosh, that's so cool. But big packet, big packet is nice. ... So you're going to be right here in Los Angeles; that is so cool. I'm looking forward to it. So can we hang out? Hey, yes, you know that's - that's what I'm coming for. Oh, yay. Thank you. No but that's an awesome school. ... Well, you know I'm leaning towards like communicationaI. Okay. They have a lot of good schools, right? Yeah, I mean it's just a really good school. I'm looking forward to getting there. I mean, God, the campus is cool. Figure 1: Illustration of an example conversation in the IEMOCAP dataset the research focus on emotion recognition in con- versation. Different from traditional emotion recog- nition on isolated utterances, emotion recognition in conversation requires context modeling of indi- vidual utterances. The context can be attributed to the preceding utterances, temporality in con- versation turns, or speaker related information etc. Different models have been proposed to capture …

Content truncated to 500 words