UniMSE: Towards Unified Multimodal Sentiment Analysis and Emotion Recognition

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7837 - 7851 December 7-11, 2022 ©2022 Association for Computational Linguistics UniMSE: Towards Unified Multimodal Sentiment Analysis and Emotion Recognition Guimin Hu†, Ting-En Lin, Yi Zhao†∗, Guangming Lu†, Yuchuan Wu, Yongbin Li∗ †School of Computer Science and Technology, School of Science, Harbin Institute of Technology (Shenzhen), China {rice.hu.x,liyb821}@gmail.com {zhao.yi,luguangm}@hit.edu.cn Abstract Multimodal sentiment analysis (MSA) and emotion recognition in conversation (ERC) are key research topics for computers to understand human behaviors. From a psychological per- spective, emotions are the expression of affect or feelings during a short period, while senti- ments are formed and held for a longer period. However, most existing works study sentiment and emotion separately and do not fully exploit the complementary knowledge behind the two. In this paper, we propose a multimodal senti- ment knowledge-sharing framework (UniMSE) that unifies MSA and ERC tasks from features, labels, and models. We perform modality fu- sion at the syntactic and semantic levels and introduce contrastive learning between modali- ties and samples to better capture the difference and consistency between sentiments and emo- tions. Experiments on four public benchmark datasets, MOSI, MOSEI, MELD, and IEMO- CAP, demonstrate the effectiveness of the pro- posed method and achieve consistent improve- ments compared with state-of-the-art methods. 1 Introduction With the rapid development of multimodal ma- chine learning (Liang et al., 2022; Baltrušaitis et al., 2018) and dialog system (He et al., 2022a,b,c), Mul- timodal Sentiment Analysis (MSA) and Emotion Recognition in Conversations (ERC) have become the keys for machines to perceive, recognize, and understand human behaviors and intents (Zhang et al., 2021a,b; Hu et al., 2021a,b). Multimodal data provides not only verbal information, such as textual (spoken words) features but also non-verbal information, including acoustic (prosody, rhythm, pitch) and visual (facial attributes) features. These different modalities allow the machine to make de- cisions from different perspectives, thereby achiev- ing more accurate predictions (Ngiam et al., 2011). The goal of MSA is to predict sentiment intensity ∗Corresponding authors. Figure 1: Illustration of sentiment and emotion sharing a unified embedding space. The bottom is a unified label after formalizing sentiment and emotion according to the similarity ⌢ sim between samples with the same sentiment polarity label. or polarity, and ERC aims to predict predefined emotion categories. There are many research direc- tions in MSA and ERC, such as multimodal fusion (Yang et al., 2021), modal alignment (Tsai et al., 2019a), context modeling (Mao et al., 2021) and external knowledge (Ghosal et al., 2020). How- ever, most existing works treat MSA and ERC as separate tasks, ignoring the similarities and com- plementarities between sentiments and emotions. On the one hand, from a psychological perspec- tive, both sentiments and emotions are experiences that result from the combined influences of the bi- ological, cognitive, and social (Stets, 2006), and could be expressed similarly. In Figure 1, we illus- trate how sentiments and emotions are relevant in the verbal or non-verbal, and could be projected into a …

Content truncated to 500 words