Tackling Modality Heterogeneity with Multi-View Calibration Network for Multimodal Sentiment Detection

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 5240–5252 July 9-14, 2023 ©2023 Association for Computational Linguistics Tackling Modality Heterogeneity with Multi-View Calibration Network for Multimodal Sentiment Detection Yiwei Wei1,4∗, Shaozu Yuan2∗†, Ruosong Yang5, Lei Shen2 Longbiao Wang1,3†, Zhangmeizhi Li4, Meng Chen2 1Tianjin Key Laboratory of Cognitive Computing and Application, College of Intelligence and Computing, Tianjin University, Tianjin, China 2JD AI Research, Beijing, China 3Huiyan Technology (Tianjin) Co., Ltd, Tianjin, China 4China University of Petroleum(Beijing) at Karamay, Karamay, China 5The Hong Kong Polytechnic University, Hongkong, China Abstract With the popularity of social media, detecting sentiment from multimodal posts (e.g. image- text pairs) has attracted substantial attention recently. Existing works mainly focus on fus- ing different features but ignore the challenge of modality heterogeneity. Specifically, dif- ferent modalities with inherent disparities may bring three problems: 1) introducing redundant visual features during feature fusion; 2) caus- ing feature shift in the representation space; 3) leading to inconsistent annotations for differ- ent modal data. All these issues will increase the difficulty in understanding the sentiment of the multimodal content. In this paper, we pro- pose a novel Multi-View Calibration Network (MVCN) to alleviate the above issues system- atically. We first propose a text-guided fusion module with novel Sparse-Attention to reduce the negative impacts of redundant visual ele- ments. We then devise a sentiment-based con- gruity constraint task to calibrate the feature shift in the representation space. Finally, we introduce an adaptive loss calibration strategy to tackle inconsistent annotated labels. Exten- sive experiments demonstrate the competitive- ness of MVCN against previous approaches and achieve state-of-the-art results on two pub- lic benchmark datasets. 1 Introduction Multimodal sentiment detection(Xu, 2017; Xu and Mao, 2017) aims to explore the sentiment embed- ded in the multimodal contents, such as text, im- ages, and videos. With the growth of social media, it shows broad applications for the understanding of one’s position, attitude, or opinion towards an entity, person, or topic, which has attracted sub- stantial attention from both academic and industrial ∗Both authors contribute equally to this work. † Corresponding authors. Email:{yuanshaozu@jd.com, longbiao wang@tju.edu.cn.} Figure 1: Examples of multimodal sentiment detection. communities. In this paper, we focus on detecting the sentiment of multimodal posts in social media. As shown in Figure 1, the model is required to in- fer the human sentiments given the image and text pairs. Previous works mainly focus on how to integrate modalities and have achieved astonishing progress. Earlier works apply concatenation to fuse different features, such as Xu (2017); Xu and Mao (2017). To promote the fusion, Yang et al. (2020); Yu and Jiang (2019); Kumar and Vepa (2020); Xu et al. (2018) construct different modules to realize deeper multimodal interaction. However, these methods of viewing different modalities in the same light neglect the modality heterogeneity, hindering the performance of the model. Although CLMLF (Li et al., 2022) tries to apply contrastive learning to alleviate this problem, it is in coarse granularity and only considers …

Content truncated to 500 words