AMOA: Global Acoustic Feature Enhanced Modal-Order-Aware Network for Multimodal Sentiment Analysis

Proceedings of the 29th International Conference on Computational Linguistics, pages 7136–7146 October 12–17, 2022. 7136 AMOA: Global Acoustic Feature Enhanced Modal-Order-Aware Network for Multimodal Sentiment Analysis Ziming Li1,2, Yan Zhou1,∗, Weibo Zhang3, Yaxin Liu1,2, Chuanpeng Yang1,2, Zheng Lian4, Songlin Hu1,2 1Institute of Information Engineering, Chinese Academy of Sciences 2School of Cyber Security, University of Chinese Academy of Sciences 3China Electronics Standardization Institute 4Institute of Automation, Chinese Academy of Sciences {liziming, zhouyan, liuyaxin, yangchuanpeng, husonglin}@iie.ac.cn zhangweibo@cesi.cn lianzheng2016@ia.ac.cn Abstract In recent years, multimodal sentiment analy- sis (MSA) has attracted more and more inter- est, which aims to predict the sentiment po- larity expressed in a video. Existing methods typically 1) treat three modal features (textual, acoustic, visual) equally, without distinguish- ing the importance of different modalities; and 2) split the video into frames, leading to miss- ing the global acoustic information. In this paper, we propose a global Acoustic feature en- hanced Modal-Order-Aware network (AMOA) to address these problems. Firstly, a modal- order-aware network is designed to obtain the multimodal fusion feature. This network inte- grates the three modalities in a certain order, which makes the modality at the core position matter more. Then, we introduce the global acoustic feature of the whole video into our model. Since the global acoustic feature and multimodal fusion feature originally reside in their own spaces, contrastive learning is further employed to align them before concatenation. Experiments on two public datasets show that our model outperforms the state-of-the-art mod- els. In addition, we also generalize our model to the sentiment with more complex seman- tics, such as sarcasm detection. Our model also achieves state-of-the-art performance on a widely used sarcasm dataset. 1 Introduction Multimodal sentiment analysis (MSA) has attracted more and more attention in recent years. In many cases, we need to combine the textual, acoustic, and visual features to predict sentiment polarity. For example, the tone of a person’s voice and the changing expression can both have an impact on sentiment polarity prediction. ∗ Corresponding author. In most previous works, each modality will go through the same process at the fusion, or in other words, the three modalities are treated equally (Hasan et al., 2021; Chauhan et al., 2020). How- ever, for sentiment analysis, the textual modality is usually the core modality based on life experi- ence and previous works (Tsai et al., 2019; Han et al., 2021a; Hasan et al., 2021), because the text contains the most basic semantic information. The acoustic feature also plays an important role: a speech with a rising tone is more likely to express positive sentiment. Finally, facial expressions also have impacts on sentiment, such as the rising range of the corners of the mouth and the size of the pupils. However, the information about sentiment in expression is not as rich as that in tone. In addi- tion, visual information does not always correspond to text like acoustic information. In many situa- tions, the change in the speaker’s facial expression is quite subtle. Even more, the speaker is …

Content truncated to 500 words