A Text-Centered Shared-Private Framework via Cross-Modal Prediction for Multimodal Sentiment Analysis

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4730–4738 August 1–6, 2021. ©2021 Association for Computational Linguistics 4730 A Text-Centered Shared-Private Framework via Cross-Modal Prediction for Multimodal Sentiment Analysis Yang Wu1 Zijie Lin1 Yanyan Zhao1∗ Bing Qin1 Li-Nan Zhu2 1 Harbin Institute of Technology 2 Zhejiang University of Technology 1 {ywu, linzijie97, yyzhao, qinb}@ir.hit.edu.cn 2 zln@zjut.edu.cn Abstract Multimodal fusion is a core problem for multi- modal sentiment analysis. Previous works usu- ally treat all three modal features equally and implicitly explore the interactions between dif- ferent modalities. In this paper, we break this kind of methods in two ways. Firstly, we ob- serve that textual modality plays the most im- portant role in multimodal sentiment analysis, and this can be seen from the previous works. Secondly, we observe that comparing to the textual modality, the other two kinds of non- textual modalities (visual and acoustic) can provide two kinds of semantics, shared and pri- vate semantics. The shared semantics from the other two modalities can obviously enhance the textual semantics and make the sentiment analysis model more robust, and the private semantics can be complementary to the tex- tual semantics and meanwhile provide differ- ent views to improve the performance of senti- ment analysis together with the shared seman- tics. Motivated by these two observations, we propose a text-centered shared-private frame- work (TCSP) for multimodal fusion, which consists of the cross-modal prediction and sen- timent regression parts. Experiments on the MOSEI and MOSI datasets demonstrate the effectiveness of our shared-private framework, which outperforms all baselines. Furthermore, our approach provides a new way to utilize the unlabeled data for multimodal sentiment anal- ysis. 1 Introduction Multimodal sentiment analysis is an emerging re- search field, which aims to understand people’s sen- timent using not only textual but also non-textual (visual, acoustic) data. This task has attracted in- creasing attention from the community recently, as people have realized that non-textual clues are help- ful for detecting sentiment and the huge demands ∗ Corresponding Author Textual Visual Acoustic Cross-Modal Prediction [Private] Higher Prediction Losses [Shared] Higher Attention Weights [Private] Higher Prediction Losses [Shared] Higher Attention Weights Figure 1: Distinguishing the shared and private features via cross-modal prediction. for the identification of opinions and sentiment in the video. Comparing to the traditional textual sentiment analysis(Liu, 2012), previous work demonstrates that the other non-textual data can improve the fi- nal performance (Chen et al., 2017; Zadeh et al., 2018b; Sun et al., 2020). There are two reasons. The first reason is that the three modalities can con- vey some common semantics. In this case, these non-textual common semantics do not provide ad- ditional information beyond textual data, but the repetitive information from them can strengthen the final performance. We call them shared semantics. The other reason is that the three modalities have their own special semantics, which are different to other modalities. These non-textual private seman- tics is modality-specific and hard to be predicted only by textual data. Thus this kind of semantics from …

Content truncated to 500 words