Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysis

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9180–9192 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 9180 Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysis Wei Han†, Hui Chen†, Soujanya Poria†, † Singapore University of Technology and Design, Singapore {wei han, hui chen}@mymail.sutd.edu.sg sporia@sutd.edu.sg Abstract In multimodal sentiment analysis (MSA), the performance of a model highly depends on the quality of synthesized embeddings. These em- beddings are generated from the upstream pro- cess called multimodal fusion, which aims to extract and combine the input unimodal raw data to produce a richer multimodal represen- tation. Previous work either back-propagates the task loss or manipulates the geometric property of feature spaces to produce favor- able fusion results, which neglects the preser- vation of critical task-related information that flows from input to the fusion results. In this work, we propose a framework named MultiModal InfoMax (MMIM), which hier- archically maximizes the Mutual Information (MI) in unimodal input pairs (inter-modality) and between multimodal fusion result and unimodal input in order to maintain task- related information through multimodal fu- sion. The framework is jointly trained with the main task (MSA) to improve the perfor- mance of the downstream MSA task. To ad- dress the intractable issue of MI bounds, we further formulate a set of computationally sim- ple parametric and non-parametric methods to approximate their truth value. Experimen- tal results on the two widely used datasets demonstrate the efficacy of our approach. The implementation of this work is pub- licly available at https://github.com/ declare-lab/Multimodal-Infomax. 1 Introduction With the unprecedented advances in social media in recent years and the availability of smartphones with high-quality cameras, we witness an explo- sive boost of multimodal data, such as movies, short-form videos, etc. In real life, multimodal data usually consists of three channels: visual (im- age), acoustic (voice), and transcribed text. Many of them often express sort of sentiment, which is a long-term disposition evoked when a person encounters a specific topic, person or entity (De- onna and Teroni, 2012; Poria et al., 2020). Min- ing and understanding these emotional elements from multimodal data, namely multimodal senti- ment analysis (MSA), has become a hot research topic because of numerous appealing applications, such as obtaining overall product feedback from customers or gauging polling intentions from po- tential voters (Melville et al., 2009). Generally, different modalities in the same data segment are often complementary to each other, providing ex- tra cues for semantic and emotional disambigua- tion (Ngiam et al., 2011). The crucial part for MSA is multimodal fusion, in which a model aims to extract and integrate information from all input modalities to understand the sentiment behind the seen data. Existing methods to learn unified repre- sentations are grouped in two categories: through loss back-propagation or geometric manipulation in the feature spaces. The former only tunes the parameters based on back-propagated gradi- ents from the task loss (Zadeh et al., 2017; Tsai et al., 2019a; Ghosal et al., 2019), reconstruction …

Content truncated to 500 words