Modeling Intra- and Inter-Modal Relations: Hierarchical Graph Contrastive Learning for Multimodal Sentiment Analysis

Proceedings of the 29th International Conference on Computational Linguistics, pages 7124–7135 October 12–17, 2022. 7124 Modeling Intra- and Inter-Modal Relations: Hierarchical Graph Contrastive Learning for Multimodal Sentiment Analysis Zijie Lin 1, Bin Liang 1∗, Yunfei Long 2, Yixue Dang 3, Min Yang 4, Min Zhang 1, Ruifeng Xu 1, 5, 6∗ 1 Harbin Institute of Technology, Shenzhen, China 2 University of Essex, Colchester, UK 3 China Merchants Securities Co ., Ltd. 4 Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences 5 Peng Cheng Laboratory, Shenzhen, China 6 Guangdong Provincial Key Laboratory of Novel Security Intelligence Technologies lzjjeffery@163.com, 18B951033@stu.hit.edu.cn xuruifeng@hit.edu.cn Abstract The existing research efforts in Multimodal Sentiment Analysis (MSA) have focused on developing the expressive ability of neural networks to fuse information from different modalities. However, these approaches lack a mechanism to understand the complex re- lations within and across different modalities, since some sentiments may be scattered in dif- ferent modalities. To this end, in this paper, we propose a novel hierarchical graph contrastive learning (HGraph-CL) framework for MSA, aiming to explore the intricate relations of intra- and inter-modal representations for sentiment extraction. Specifically, regarding the intra- modal level, we build a unimodal graph for each modality representation to account for the modality-specific sentiment implications. Based on it, a graph contrastive learning strat- egy is adopted to explore the potential relations based on unimodal graph augmentations. Fur- thermore, we construct a multimodal graph of each instance based on the unimodal graphs to grasp the sentiment relations between different modalities. Then, in light of the multimodal augmentation graphs, a graph contrastive learn- ing strategy over the inter-modal level is pro- posed to ulteriorly seek the possible graph struc- tures for precisely learning sentiment relations. This essentially allows the framework to under- stand the appropriate graph structures for learn- ing intricate relations among different modal- ities. Experimental results on two benchmark datasets show that the proposed framework out- performs the state-of-the-art baselines in MSA. 1 Introduction Multimodal Sentiment Analysis (MSA) has re- ceived increasing research attention in recent years. Different from textual sentiment analysis, MSA generally contains three modalities: text (caption), * Corresponding author. visual text acoustic strong weak … … (shake head & roll eyes) (frown) He was the only character that was slightly interesting Figure 1: An example of intricate relations among dif- ferent modalities in MSA. visual and acoustic, in which the visual and acous- tic information can complement the text for iden- tifying the sentiment score and thus preferably de- tecting the sentiment polarity (e.g. positive, neg- ative, etc.). As shown in Figure 1, although the text modality expresses the positive sentiment, we can infer that the correct sentiment of this exam- ple is negative in the light of the study of visual and acoustic information. Therefore, dealing with MSA needs to consider learning and fusing the information from different modalities. Early MSA work attempted to fuse the infor- mation from different modalities by tensor-based features fusion (Snoek et al., 2005; Zadeh et al., 2017; Liu …

Content truncated to 500 words