Multimodal fusion via cortical network inspired losses

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 1167 - 1178 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Multimodal fusion via cortical network inspired losses Shiv Shankar University of Massachusetts sshankar@cics.umass.edu Abstract Information integration from different modal- ities is an active area of research. Human be- ings and, in general, biological neural systems are quite adept at using a multitude of signals from different sensory perceptive fields to in- teract with the environment and each other. Re- cent work in deep fusion models via neural networks has led to substantial improvements over unimodal approaches in areas like speech recognition, emotion recognition and analysis, captioning and image description. However, such research has mostly focused on architec- tural changes allowing for fusion of different modalities while keeping the model complexity manageable. Inspired by neuroscientific ideas about multisensory integration and processing, we investigate the effect of introducing neu- ral dependencies in the loss functions. Experi- ments on multimodal sentiment analysis tasks with different models show that our approach provides a consistent performance boost. 1 Introduction Human beings perceive the world as a unified whole, not in individual sensory modalities. While traditionally different sensory models have been studied in isolation, it has been well recognized that perception operates via integration of informa- tion from multiple sensory modalities. Research in multimodal fusion aims to achieve a similar goal in artificial models: extract and in- tegrate all information from different input modal- ities. For example, if someone is sarcastic, the facial expression and voice intonation provide in- formation not directly decipherable from the ut- tered words. If a model only looks at the text of the interaction, then it is unlikely to classify this interaction currently. Current research in deep mul- timodal fusion primarily deals with architectural improvements to create complex feature-rich, yet efficient representations (Zadeh et al., 2017; Liu et al., 2018; Hazarika et al., 2020). The hope is that more complex models will be able to integrate the complementary information from different uni- modal representations into a unified common rep- resentation. Learning such unified representations, however, is a challenging task. Different modalities can present the same information in radically dif- ferent ways with emphasis on different aspects of the content. These heterogeneities across different modalities mean that learning multimodal represen- tations must deal with feature shifts, distributional effects, nuisance variation and a variety of related challenges (Baltrušaitis et al., 2018). Inspiring from work in multisensory neural pro- cessing, we define a loss regularizer that we call synergy to train these models. Synergy has a spe- cific meaning in information-theoretic literature (Cover, 1999). The synergy between random vari- ables X and Y refers to the unique mutual infor- mation that X provides about Y . While our loss function is not the same as information theoretic synergy, the intuition behind our proposed loss is the same as actual synergy; to try to maximize de- pendencies between the representations. As our method uses neural networks or …

Content truncated to 500 words