SWAFN: Sentimental Words Aware Fusion Network for Multimodal Sentiment Analysis

Proceedings of the 28th International Conference on Computational Linguistics, pages 1067–1077 Barcelona, Spain (Online), December 8-13, 2020 1067 SWAFN: Sentimental Words Aware Fusion Network for Multimodal Sentiment Analysis Minping Chen, Xia Li∗ Guangzhou Key Laboratory of Multilingual Intelligent Processing, School of Information Science and Technology, Guangdong University of Foreign Studies,Guangzhou,China {minpingchen,xiali}@gdufs.edu.cn Abstract Multimodal sentiment analysis aims to predict sentiment of language text with the help of other modalities, such as vision and acoustic features. Previous studies focused on learning the joint representation of multiple modalities, ignoring some useful knowledge contained in language modal. In this paper, we try to incorporate sentimental words knowledge into the fusion net- work to guide the learning of joint representation of multimodal features. Our method consists of two components: shallow fusion part and aggregation part. For the shallow fusion part, we use crossmodal coattention mechanism to obtain bidirectional context information of each two modals to get the fused shallow representations. For the aggregation part, we design a multitask of sentimental words classification to help and guide the deep fusion of the three modalities and obtain the final sentimental words aware fusion representation. We carry out several experiments on CMU-MOSI, CMU-MOSEI and YouTube datasets. The experimental results show that intro- ducing sentimental words prediction as a multitask can really improve the fusion representation of multiple modalities. 1 Introduction Multimodal sentiment analysis is a task of predicting the sentiment of a video, an image or a text based on multiple modal features. Based on the contributions of different modalities to each other, multimodal sentiment analysis has achieved significant results and attracted the attentions of many researchers in recent years. The main challenge of the multimodal sentiment analysis is to capture a better fusion of different modalities. Previous studies have proposed different methods for the fusion in different point of views. Some methods focus on the improvement of the LSTM structure to learn the interactions of different modal features from the view of the uni-stage and multi-stage. Zadeh et al. (2018a) propose a Memory Fusion Network to learn both the view-specific interactions and the cross-view interactions. Liang et al. (2018) propose a Recurrent Multistage Fusion Network to model cross-modal interactions using multi- stage fusion approach. Some methods focus on exploiting the expressiveness of tensors for multimodal representation. Zadeh et al. (2017) propose a Tensor Fusion Network to explicitly model the unimodal, bimodal and trimodal interactions through a 3-fold Cartesian product from modality embedding. More recently, other methods are proposed and achieve new state-of-the-art results (Pham et al., 2019; Mai et al., 2019; Wang et al., 2019; Tsai et al., 2019). Although previous studies achieved good results, there are still two points can be improved: (1) We find that the fusion of most of previous methods is from one direction, that is, when the two modals are fused, the representation of two modals are fused directly as a new representation, similar to the work of Zadeh et al. (2017) and Liu et al. (2018). This fusion strategy ignores the …

Content truncated to 500 words