Analyzing Modality Robustness in Multimodal Sentiment Analysis

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 685 - 696 July 10-15, 2022 ©2022 Association for Computational Linguistics Analyzing Modality Robustness in Multimodal Sentiment Analysis Devamanyu Hazarika ∗, Yingting Liα DeCLaRe∗, Bo Chengα, Shuai Zhaoα, Roger Zimmermann , Soujanya Poria DeCLaRe National University of Singapore, Singapore α Beijing University of Posts and Telecommunications DeCLaReDeCLaRe Lab, Singapore University of Technology and Design, Singapore {hazarika, rogerz}@comp.nus.edu.sg {cindyyting, chengbo, zhaoshuaiby}@bupt.edu.cn sporia@sutd.edu.sg Abstract Building robust multimodal models are cru- cial for achieving reliable deployment in the wild. Despite its importance, less attention has been paid to identifying and improving the robustness of Multimodal Sentiment Anal- ysis (MSA) models. In this work, we hope to address that by (i) Proposing simple di- agnostic checks for modality robustness in a trained multimodal model. Using these checks, we find MSA models to be highly sensitive to a single modality, which creates issues in their robustness; (ii) We analyze well-known robust training strategies to alleviate the is- sues. Critically, we observe that robustness can be achieved without compromising on the original performance. We hope our extensive study–performed across five models and two benchmark datasets–and proposed procedures would make robustness an integral component in MSA research. Our diagnostic checks and robust training solutions are simple to imple- ment and available at https://github.com/ declare-lab/MSA-Robustness 1 Introduction Multimodal Sentiment Analysis (MSA) is a bur- geoning field of research that has seen accelerated developments in recent years. Numerous models have been proposed that utilize multiple modalities such as audio, visual, and language signals to pre- dict sentiments, emotions, and other forms of affect. While progress in MSA has been driven mainly by improvements in multimodal performance, we call for attention towards an equally important aspect ∗Equal Contribution. Regression - Correlation -0.2 0 0.2 0.4 0.6 0.8 MISA (-) Text (-) Visual (-) Audio 0.739 0.74 0.001 0.737 Classification - F1 0 22.5 45 67.5 90 MISA (-) Text (-) Visual (-) Audio 81.85 82.34 27.98 82.4 BBFN (-) Text (-) Visual (-) Audio 0.754 0.755 0.023 0.754 Self-MM (-) Text (-) Visual (-) Audio 0.794 0.794 0.049 0.794 MMIM 0.796 MISA BBFN Self-MM Figure 1: Removing modalities one at a time from the test- ing set of CMU-MOSI (Zadeh et al., 2016) on a trained MISA (Hazarika et al., 2020). in multimodal systems – multimodal robustness. Robustness is crucial when models are deployed in the wild, where it is common to encounter inad- vertent errors in the source modalities due to data loss, data corruption, jitter, privacy issues, amongst others. A well-known fact in the MSA research is that language modality tends to be the most effective, which has prompted models to utilize language as its core modality (Wu et al., 2021; Han et al., 2021a; Zeng et al., 2021). In this work, we focus on skewed dependence on language and try to under- stand how it affects the robustness of MSA models. Specifically, we ask, RQ1: Are models …

Content truncated to 500 words