Mitigating Inconsistencies in Multimodal Sentiment Analysis under Uncertain Missing Modalities

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2924 - 2934 December 7-11, 2022 ©2022 Association for Computational Linguistics Mitigating Inconsistencies in Multimodal Sentiment Analysis under Uncertain Missing Modalities Jiandian Zeng1, Jiantao Zhou1∗and Tianyi Liu2 1 State Key Laboratory of Internet of Things for Smart City 1 Department of Computer and Information Science, University of Macau 2 Department of Computer Science and Engineering, Shanghai Jiao Tong University {yb87470, jtzhou}@um.edu.mo, liutianyi@sjtu.edu.cn Abstract For the missing modality problem in Multi- modal Sentiment Analysis (MSA), the incon- sistency phenomenon occurs when the senti- ment changes due to the absence of a modal- ity. The absent modality that determines the overall semantic can be considered as a key missing modality. However, previous works all ignored the inconsistency phenomenon, simply discarding missing modalities or solely gener- ating associated features from available modal- ities. The neglect of the key missing modality case may lead to incorrect semantic results. To tackle the issue, we propose an Ensemble-based Missing Modality Reconstruction (EMMR) net- work to detect and recover semantic features of the key missing modality. Specifically, we first learn joint representations with remaining modalities via a backbone encoder-decoder net- work. Then, based on the recovered features, we check the semantic consistency to determine whether the absent modality is crucial to the overall sentiment polarity. Once the inconsis- tency problem due to the key missing modality exists, we integrate several encoder-decoder approaches for better decision making. Exten- sive experiments and analyses are conducted on CMU-MOSI and IEMOCAP datasets, vali- dating the superiority of the proposed method. 1 Introduction Sentiment analysis has witnessed significant progress in the past years (Zhang et al., 2016), where the traditional textual sentiment classifica- tion has developed into more complex Multimodal Sentiment Analysis (MSA) models. Taking the phase “Yeah, I think so.” for instance, it is hard to read the emotion without enough lexical infor- mation, and the acoustic modality may help in the emotion recognition if available. Thus, it is cru- cial to combine different modalities together for accurate sentiment analysis. ∗ Corresponding Author Modality Content True Predict Neutral (√) Negative (×) Acoustic (slight, neutral) Visual Textual I actually am not finding the series to be very funny and popcorn kind of movie. Figure 1: Case of missing the key modality, where the missing modality is marked with dotted red lines, and the semantic words are marked in blue. So far, MSA has been well studied under the assumption that all modalities are always available. However, in reality, such a strong assumption does not always hold, and we often encounter scenarios that partial modalities could be missing. To address the missing data problem, a consequent effort has been made on recovering absent modalities. Tran et al. (2017) first identified the missing modality problem in multimodal data. More recently, several works (Suo et al., 2019; Ma et al., 2021; Zhao et al., 2021; Yuan et al., 2021; Zeng et al., 2022) focused on the missing modalities problem in an uncertain …

Content truncated to 500 words