Missing Modality meets Meta Sampling (M3S): An Efficient Universal Approach for Multimodal Sentiment Analysis with Missing Modality

Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 121–130 November 20–23, 2022. ©2022 Association for Computational Linguistics 121 Missing Modality meets Meta Sampling (M3S): An Efficient Universal Approach for Multimodal Sentiment Analysis with Missing Modality Haozhe Chi Minghua Yang Junhao Zhu Guanhong Wang Gaoang Wang∗ Zhejiang University-University of Illinois at Urbana-Champaign Institute, Zhejiang University, China {haozhe.20, minghua.20, junhao.20, gaoangwang}@intl.zju.edu.cn guanhongwang@zju.edu.cn Abstract Multimodal sentiment analysis (MSA) is an important way of observing mental activities with the help of data captured from multiple modalities. However, due to the recording or transmission error, some modalities may in- clude incomplete data. Most existing works that address missing modalities usually assume a particular modality is completely missing and seldom consider a mixture of missing across multiple modalities. In this paper, we propose a simple yet effective meta-sampling approach for multimodal sentiment analysis with miss- ing modalities, namely Missing Modality-based Meta Sampling (M3S). To be specific, M3S for- mulates a missing modality sampling strategy into the modal agnostic meta-learning (MAML) framework. M3S can be treated as an efficient add-on training component on existing models and significantly improve their performances on multimodal data with a mixture of miss- ing modalities. We conduct experiments on IEMOCAP, SIMS and CMU-MOSI datasets, and superior performance is achieved compared with recent state-of-the-art methods. 1 Introduction Multimodal sentiment analysis (MSA) aims to esti- mate human mental activities by multimodal data, such as a combination of audio, video, and text. Though much progress has been made recently, there still exist challenges, including missing modal- ity problem. In reality, missing modality is a com- mon problem due to the errors in data collection, storage, and transmission. To address the issue with missing modality in MSA, many approaches have been proposed (Ma et al., 2021c; Zhao et al., 2021; Ma et al., 2021b; Parthasarathy and Sundaram, 2020; Ma et al., 2021a; Tran et al., 2017). In general, methods that address the missing modality issue usually only consider the situation where a certain input modality is severely damaged. ∗Corresponding author. Figure 1: M3S helps MMIN model achieve superior performance. The strategies of these proposed methods can be divided into three categories: 1) Designing new ar- chitectures with a reconstruction network to recover missing modality with the information from other modalities (Ma et al., 2021c; Ding et al., 2014); 2) Formulating innovative and efficient loss functions to tackle missing modality (Ma et al., 2021a, 2022); 3) Improving the encoding and embedding strate- gies from existing models (Tran et al., 2017; Cai et al., 2018). In the MSA tasks, most of the proposed methods focus on the situation where certain modalities are completely missing and the other modalities are complete. However, due to the transmission or collection errors, each modality may contain partial information based on a certain missing rate, while existing methods seldom consider this type of scenario and they are not suitable to be …

Content truncated to 500 words