Multimodal Sentiment Analysis with Multi-perspective Fusion Network Focusing on Sense Attentive Language

Multimodal Sentiment Analysis with Multi-perspective Fusion Network Focusing on Sense Attentive Language Xia Li* , Minping Chen Guangzhou Key Laboratory of Multilingual Intelligent Processing, School of Information Science and Technology, Guangdong University of Foreign Studies,Guangzhou,China {xiali,minpingchen}@gdufs.edu.cn Abstract Multimodal sentiment analysis aims to learn a joint representation of multiple features. As demonstrated by previous studies, it is shown that the language modality may contain more semantic information than that of other modalities. Based on this observation, we propose a Multi-perspective Fusion Network(MPFN) focusing on Sense Attentive Language for multi- modal sentiment analysis. Different from previous studies, we use the language modality as the main part of the final joint representation, and propose a multi-stage and uni-stage fusion strategy to get the fusion representation of the multiple modalities to assist the final language-dominated multimodal representation. In our model, a Sense-Level Attention Network is proposed to dy- namically learn the word representation which is guided by the fusion of the multiple modalities. As in turn, the learned language representation can also help the multi-stage and uni-stage fusion of the different modalities. In this way, the model can jointly learn a well integrated final rep- resentation focusing on the language and the interactions between the multiple modalities both on multi-stage and uni-stage. Several experiments are carried on the CMU-MOSI, the CMU- MOSEI and the YouTube public datasets. The experiments show that our model performs better or competitive results compared with the baseline models. 1 Introduction Multimodal sentiment analysis is a task of predicting sentiment of a video, an image or a text based on multiple modal features. With the increase of short videos on the internet, such as Douyin, YouTube, etc., multimoal sentiment analysis can be used to analyze the opinions of the public based on the speaker’s language, facial gestures and acoustic behaviors. Based on the successes in video, image, audio and language processing, multimodal sentiment analysis has been studied extensively and produced impressive results in recent years (Liang et al., 2018; Liu et al., 2018; Mai et al., 2019; Wang et al., 2019; Zadeh et al., 2018c). The core of the multimodal sentiment analysis is to capture a better fusion of different modalities. Different methods are proposed to fuse the multimodal features and help to capture the interactions of the modalities. Tensor Fusion Network (Zadeh et al., 2017) is proposed to obtain raw unimodal representations, bimodal interactions and tri- modal interactions in the form of 2D-tensor and 3D-tensor simultaneously. Low-rank Fusion Network (Liu et al., 2018) is then proposed to alleviate the drawback of the large amount of parameters by low- rank factor. Although the above methods achieved good results, they treat all modalities equally and fuse the modalities in the same contribution. We find that language modality always contain more semantic information for sentiment analysis, that’s why most of ablation experiments of previous studies (Mai et al., 2019; Poria et al., 2017a; Zadeh et al., 2017) show that when using features from only one modality, the model using language features performs …

Content truncated to 500 words