MMDAG: Multimodal Directed Acyclic Graph Network for Emotion Recognition in Conversation

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 6802–6807 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 6802 MMDAG: Multimodal Directed Acyclic Graph Network for Emotion Recognition in Conversation Shuo Xu1,2, Yuxiang Jia1*, Changyong Niu1, Hongying Zan1 1. School of Computer and Artificial Intelligence, Zhengzhou University, China 2. Zhengzhou Zoneyet Technology Co., Ltd., Zhengzhou, China zzunlpxs1306@163.com, {ieyxjia, iecyniu, iehyzan}@zzu.edu.cn Abstract Emotion recognition in conversation is important for an empathetic dialogue system to understand the user’s emotion and then generate appropriate emotional responses. However, most previous researches focus on modeling conversational contexts primarily based on the textual modality or simply utilizing multimodal information through feature concatenation. In order to exploit multimodal information and contextual information more effectively, we propose a multimodal directed acyclic graph (MMDAG) network by injecting information flows inside modality and across modalities into the DAG architecture. Experiments on IEMOCAP and MELD show that our model outperforms other state-of-the-art models. Comparative studies validate the effectiveness of the proposed modality fusion method. Keywords: Multimodal, Directed Acyclic Graph Network, Emotion Recognition in Conversation 1. Introduction Emotion can influence how a person thinks, feels and behaves, and it is an important part of human daily life. The Emotion Recognition in Conversation (ERC) task aims to identify the emotion of each utterance in a conversation, which draws increasing attention from researchers due to its potential applications in many domains, such as emotional conversation generation, sentiment analysis in social media (Chatterjee et al., 2019), psychoanalytic diagnostics, and understanding students’ frustration in education (Poria et al., 2019b). The rapid growth of conversation data on major so- cial media is another drive of the popularity of the ERC task. Especially those open-sourced datasets, such as IEMOCAP (Busso et al., 2008), MELD (Poria et al., 2019a), and many others, greatly promote the research of ERC. Researchers have proposed different models to uti- lize the contextual information of a conversation, such as DialogueRNN (Majumder et al., 2019) and Dia- logueGCN (Ghosal et al., 2020b). Basically, these models are usually classified into two categories: recurrence-based models and graph-based models. For recurrence-based models, the utterances are usually temporally encoded to consider distant utterances and sequential information while limited information from temporally nearest utterances makes it difficult to get a satisfactory performance. For the graph-based models, the information of the surrounding utterances within a certain window is concurrently gathered while dis- tant utterances and sequential information is usually ig- nored. DAG-ERC (Shen et al., 2021b) integrates ad- vantages of both recurrence-based model and graph- based model by representing a conversation as a di- rected acyclic graph (DAG) and proposes a DAG net- *Corresponding author work to gather information from both neighboring ut- terances and remote utterances, which achieves state- of-the-art (SOTA) performance on several benchmarks. However, DAG-ERC only uses textual modality infor- mation while acoustic and visual information has also been proven useful for ERC. Figure 1: A directed acyclic graph of a conversation, with the brown edges representing the propagation …

Content truncated to 500 words