A Cross-Modality Context Fusion and Semantic Refinement Network for Emotion Recognition in Conversation

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 13099–13110 July 9-14, 2023 ©2023 Association for Computational Linguistics CMCF-SRNet: A Cross-Modality Context Fusion and Semantic Refinement Network for Emotion Recognition in Conversation Xiaoheng Zhang Beihang University xiaoheng_zhang@buaa.edu.cn Yang Li ∗ Beihang University liyang@buaa.edu.cn Abstract Emotion recognition in conversation (ERC) has attracted enormous attention for its applications in empathetic dialogue systems. However, most previous researches simply concatenate multi- modal representations, leading to an accumu- lation of redundant information and a limited context interaction between modalities. Fur- thermore, they only consider simple contextual features ignoring semantic clues, resulting in an insufficient capture of the semantic coherence and consistency in conversations. To address these limitations, we propose a cross-modality context fusion and semantic refinement net- work (CMCF-SRNet). Specifically, we first design a cross-modal locality-constrained trans- former to explore the multimodal interaction. Second, we investigate a graph-based seman- tic refinement transformer, which solves the limitation of insufficient semantic relationship information between utterances. Extensive ex- periments on two public benchmark datasets show the effectiveness of our proposed method compared with other state-of-the-art methods, indicating its potential application in emotion recognition. Our model is available at https: //github.com/zxiaohen/CMCF-SRNet. 1 Introduction Emotion recognition in conversation (ERC) plays an important role in affective dialogue systems, aiming to understand and generate empathetic re- sponses (Raamkumar and Yang, 2022). Most stud- ies on ERC focus primarily on the textual modal- ity (Majumder et al., 2019). Although they can be easily extended to multimodal paradigms by per- forming early or late fusion (Poria et al., 2017a), it is difficult to capture contextual interactions be- tween modalities, which limits the utilization of multiple modalities. For instance, a tensor fu- sion network based on the utterance-level explicit alignment learning both intra-modality and inter- modality interactions via the Cartesian product was ∗Corresponding author Fig. 1. An example conversation between two speakers with corresponding emotions evoked for each utterance illustrating the importance of local context. firstly created (Zadeh et al., 2017), then a low-rank multimodal fusion network to improve efficiency and reduce trainable parameters was designed (Liu et al., 2018). A conversational memory network aligns features from different modalities by fusing multi-view information (Hazarika et al., 2018b). In addition, a cross-modal transformer was integrated that learns attention between two-modal features, thus enabling implicit enhancement of the target modality (Tsai et al., 2019). A multimodal fusion graph convolutional network for ERC was put for- ward discussing the impact of fusion methods of various modalities (Hu et al., 2021). However, these methods mostly use a simple concatenation ignoring complex interactions be- tween modalities, resulting in leveraging context information insufficiently or the problem of data sparseness. Besides, they simply consider the emo- tional impact of context in the whole conversation but neglect the emotional inertia of speakers and the fact that the local context may have a higher impact than long-distance utterances. As both the current utterance and the surround- ing contexts are vital for the emotion perception, previous works …

Content truncated to 500 words