ConFEDE: Contrastive Feature Decomposition for Multimodal Sentiment Analysis

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 7617–7630 July 9-14, 2023 ©2023 Association for Computational Linguistics ConFEDE: Contrastive Feature Decomposition for Multimodal Sentiment Analysis Jiuding Yang ∗1, Yakun Yu ∗1, Di Niu 1, Weidong Guo †2, Yu Xu 2 1University of Alberta 2Platform and Content Group, Tencent 1{jiuding,yakun2,dniu}@ualberta.ca 2{weidongguo,henrysxu}@tencent.com Abstract Multimodal Sentiment Analysis aims to pre- dict the sentiment of video content. Recent re- search suggests that multimodal sentiment anal- ysis critically depends on learning a good rep- resentation of multimodal information, which should contain both modality-invariant repre- sentations that are consistent across modalities as well as modality-specific representations. In this paper, we propose ConFEDE, a unified learning framework that jointly performs con- trastive representation learning and contrastive feature decomposition to enhance representa- tion of multimodal information. It decomposes each of the three modalities of a video sam- ple, including text, video frames, and audio, into a similarity feature and a dissimilarity fea- ture, which are learned by a contrastive relation centered around text. We conducted extensive experiments on CH-SIMS, MOSI and MOSEI to evaluate various state-of-the-art multimodal sentiment analysis methods. Experimental re- sults show that ConFEDE outperforms all base- lines on these datasets on a range of metrics. 1 Introduction Multimodal deep learning involves interpreting and analyzing multimodal signals together, where each modality refers to a way in which something is experienced and felt, e.g., the visual, audio, or lan- guage modality. With the widespread popularity of online social media, such as Instagram, Tik- Tok, Facebook, etc., videos containing multiple modalities have become a major information car- rier, which brings new challenges to content recom- mendation and classification, e.g., video question answering (Lei et al., 2021; Li et al., 2020), video captioning (Ging et al., 2020; Li et al., 2020), and video retrieval (Akbari et al., 2021; Lei et al., 2021). While traditional sentiment analysis is mainly based on language, multimodal sentiment analysis ∗These authors contributed equally to this work. †Corresponding author. (MSA) predicts the human emotion by utilizing ex- tra information available in visual and audio modal- ities of the content to assist with language-based prediction. Here, the text modality contains the semantic meaning of the spoken language. The visual modality extracts the facial characteristics (e.g., head orientation, facial expressions, and pose) of the speaker. The audio modality reflects the em- phasis on the utterance (e.g., through pitch, band- width and intensity). MSA has recently gained much attention in research for several reasons. On one hand, because of the abundance of social me- dia content, commercial interests are switching from gauging user opinions/emotions from text only to more thorough multimodal analysis based on videos. On the other hand, short video platforms (e.g., TikTok, Instagram) allow users to easily cre- ate multimodal content including visual informa- tion, audio, and inserted text, while these modali- ties are sometimes noisy or even contradicting each other in sentiments. Therefore, the presence of multimodal information in addition to the text or language itself …

Content truncated to 500 words