Denoising Bottleneck with Mutual Information Maximization for Video Multimodal Fusion

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 2231–2243 July 9-14, 2023 ©2023 Association for Computational Linguistics Denoising Bottleneck with Mutual Information Maximization for Video Multimodal Fusion Shaoxiang Wu1, Damai Dai1, Ziwei Qin1, Tianyu Liu2, Binghuai Lin2, Yunbo Cao2, Zhifang Sui1 1MOE Key Lab of Computational Linguistics, Peking University 2Tencent Cloud AI wushaoxiang@stu.pku.edu.cn {daidamai,szf}@pku.edu.cn Abstract Video multimodal fusion aims to integrate mul- timodal signals in videos, such as visual, au- dio and text, to make a complementary predic- tion with multiple modalities contents. How- ever, unlike other image-text multimodal tasks, video has longer multimodal sequences with more redundancy and noise in both visual and audio modalities. Prior denoising methods like forget gate are coarse in the granularity of noise filtering. They often suppress the redundant and noisy information at the risk of losing critical information. Therefore, we propose a denoising bottleneck fusion (DBF) model for fine-grained video multimodal fusion. On the one hand, we employ a bottleneck mechanism to filter out noise and redundancy with a re- strained receptive field. On the other hand, we use a mutual information maximization mod- ule to regulate the filter-out module to pre- serve key information within different modali- ties. Our DBF model achieves significant im- provement over current state-of-the-art base- lines on multiple benchmarks covering multi- modal sentiment analysis and multimodal sum- marization tasks. It proves that our model can effectively capture salient features from noisy and redundant video, audio, and text inputs. The code for this paper is publicly available at https://github.com/WSXRHFG/DBF. 1 Introduction With the rapid development of social platforms and digital devices, more and more videos are flooding our lives, which leads video multimodal fusion an increasingly popular focus of NLP research. Video multimodal fusion aims to integrate the information from two or more modalities (e.g., visual and audio signals) into text for a more comprehensive reason- ing. For example, multimodal sentiment analysis (Poria et al., 2020) utilizes contrast between tran- script and expression to detect sarcam, multimodal summarization (Sanabria et al., 2018) complete summary with information only exists in visual signal. However, as shown in the Figure 1, there ex- ist plenty of redundancy and noise in video mul- timodal fusion: 1) high similarity across consec- utive frames brings video redundancy; 2) useless information, such as the distracting background, in- troduces frame noise; 3) weak alignment between visual stream and text also introduces misalignment noise. To alleviate the problem of redundancy and noise in video multimodal fusion, Liu et al. (2020) control the flow of redundant and noisy informa- tion between multimodal sequences by a fusion forget gate. The fusion forget gate impairs the im- pact of noise and redundancy in a coarse grain of the whole modality, so it will also filter out some representative information in the filtered modality. In order to remove noise and redundancy while preserving critical information in video multimodal fusion, we propose a denoising fusion bottleneck (DBF) model with mutual information maximiza- tion (MI-Max). …

Content truncated to 500 words