A Facial Expression-Aware Multimodal Multi-task Learning Framework for Emotion Recognition in Multi-party Conversations

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 15445–15459 July 9-14, 2023 ©2023 Association for Computational Linguistics A Facial Expression-Aware Multimodal Multi-task Learning Framework for Emotion Recognition in Multi-party Conversations Wenjie Zheng1, Jianfei Yu1∗, Rui Xia1∗, and Shijin Wang2,3 1School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing, China 2iFLYTEK AI Research (Central China) 3State Key Laboratory of Cognitive Intelligence, Hefei, China 1{wjzheng, jfyu, rxia}@njust.edu.cn, 23sjwang3@iflytek.com Abstract Multimodal Emotion Recognition in Multi- party Conversations (MERMC) has recently at- tracted considerable attention. Due to the com- plexity of visual scenes in multi-party conver- sations, most previous MERMC studies mainly focus on text and audio modalities while ignor- ing visual information. Recently, several works proposed to extract face sequences as visual features and have shown the importance of vi- sual information in MERMC. However, given an utterance, the face sequence extracted by previous methods may contain multiple peo- ple’s faces, which will inevitably introduce noise to the emotion prediction of the real speaker. To tackle this issue, we propose a two-stage framework named Facial expression- aware Multimodal Multi-Task learning (Fa- cialMMT). Specifically, a pipeline method is first designed to extract the face sequence of the real speaker of each utterance, which consists of multimodal face recognition, unsupervised face clustering, and face matching. With the extracted face sequences, we propose a multi- modal facial expression-aware emotion recog- nition model, which leverages the frame-level facial emotion distributions to help improve utterance-level emotion recognition based on multi-task learning. Experiments demonstrate the effectiveness of the proposed FacialMMT framework on the benchmark MELD dataset. The source code is publicly released at https: //github.com/NUSTM/FacialMMT. 1 Introduction Multimodal Emotion Recognition in Multi-party Conversations (MERMC) is a challenging task in the field of multimodal research. The complexity of the task arises from the dynamic and spontaneous nature of human communication in multi-party con- versations, which often involves multiple people expressing a variety of emotions simultaneously. In this task, the use of multiple modalities (e.g., ∗Corresponding authors. Methods #Col #Seg #Rec MELD (Poria et al., 2019) ✗ ✗ ✗ UniMSE (Hu et al., 2022b) ✗ ✗ ✗ MMGCN (Hu et al., 2021) ✓ ✗ ✗ MESM (Dai et al., 2021) ✓ ✗ ✗ M3ED (Zhao et al., 2022b) ✓ ✗ ✗ FacialMMT (Ours) ✓ ✓ ✓ Table 1: Comparison between different models for face se- quence extraction in the MERMC task. #Col represents col- lection of all possible speakers’ face sequences; #Seg rep- resents speaker segmentation, aiming to distinguish speaker sequences; #Rec represents speaker recognition, aiming to identify the real speaker. text, audio, and vision) is essential as it allows for a more comprehensive understanding of the emo- tions being expressed. Among different modalities, visual information usually plays a crucial role as it often provides direct clues for emotion prediction. For example, in Figure 1, without the information from the visual modality, it is hard to determine the anger emotion of the real speaker, i.e., Chandler. In the …

Content truncated to 500 words