MultiEMO: An Attention-Based Correlation-Aware Multimodal Fusion Framework for Emotion Recognition in Conversations

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 14752–14766 July 9-14, 2023 ©2023 Association for Computational Linguistics MultiEMO: An Attention-Based Correlation-Aware Multimodal Fusion Framework for Emotion Recognition in Conversations Tao Shi Tsinghua Shenzhen International Graduate School, Tsinghua University shitao21@mails.tsinghua.edu.cn Shao-Lun Huang∗ Tsinghua Shenzhen International Graduate School, Tsinghua University shaolun.huang@sz.tsinghua.edu.cn Abstract Emotion Recognition in Conversations (ERC) is an increasingly popular task in the Natural Language Processing community, which seeks to achieve accurate emotion classifications of utterances expressed by speakers during a con- versation. Most existing approaches focus on modeling speaker and contextual information based on the textual modality, while the com- plementarity of multimodal information has not been well leveraged, few current methods have sufficiently captured the complex correla- tions and mapping relationships across differ- ent modalities. Furthermore, existing state-of- the-art ERC models have difficulty classifying minority and semantically similar emotion cate- gories. To address these challenges, we propose a novel attention-based correlation-aware mul- timodal fusion framework named MultiEMO, which effectively integrates multimodal cues by capturing cross-modal mapping relation- ships across textual, audio and visual modal- ities based on bidirectional multi-head cross- attention layers. The difficulty of recognizing minority and semantically hard-to-distinguish emotion classes is alleviated by our proposed Sample-Weighted Focal Contrastive (SWFC) loss. Extensive experiments on two benchmark ERC datasets demonstrate that our MultiEMO framework consistently outperforms existing state-of-the-art approaches in all emotion cat- egories on both datasets, the improvements in minority and semantically similar emotions are especially significant. 1 Introduction Emotion Recognition in Conversations (ERC) is an emerging task in the field of Natural Language Processing (NLP), which aims to identify the emo- tion of each utterance in a conversation based on textual, audio and visual cues of the speaker. ERC has attracted an enormous amount of attention from both academia and industry, due to its widespread ∗Corresponding author. potentials in social media analysis (Chatterjee et al., 2019), health care services (Hu et al., 2021b), em- pathetic systems (Jiao et al., 2020), and so on. To solve the problem of ERC, numerous ap- proaches have been proposed. The majority of existing works concentrate on modeling speaker dependencies and conversational contexts (Poria et al., 2017; Hazarika et al., 2018a,c; Majumder et al., 2019; Ghosal et al., 2019, 2020; Shen et al., 2021; Hu et al., 2021a,b; Li et al., 2021a; Joshi et al., 2022; Lee and Lee, 2022), while there still exist several unsolved challenges: (1) The com- plementarity of multimodal information has not been well exploited. Apart from rich information contained in the textual modality, the tone and into- nation of the speaker can indicate the intensity of the emotion, facial expressions of interlocutors are also able to explicitly reveal emotional tendencies (Li et al., 2022). Figure 1 shows an example where the complementarity of acoustic and visual signals in addition to the textual modality is essential for an accurate emotion classification. Nevertheless, most existing approaches focus on the textual modality of utterances or simply utilize feature concatena- tion as …

Content truncated to 500 words