Adaptive Fusion Techniques for Multimodal Data

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 3156–3166 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 3156 Adaptive Fusion Techniques for Multimodal Data Gaurav Sahu, Olga Vechtomova University of Waterloo {gsahu, ovechtom}@uwaterloo.ca Abstract Effective fusion of data from multiple modal- ities, such as video, speech, and text, is chal- lenging due to the heterogeneous nature of multimodal data. In this paper, we propose adaptive fusion techniques that aim to model context from different modalities effectively. Instead of defining a deterministic fusion oper- ation, such as concatenation, for the network, we let the network decide “how” to combine a given set of multimodal features more effec- tively. We propose two networks: 1) Auto- Fusion, which learns to compress information from different modalities while preserving the context, and 2) GAN-Fusion, which regular- izes the learned latent space given context from complementing modalities. A quantita- tive evaluation on the tasks of multimodal ma- chine translation and emotion recognition sug- gests that our lightweight, adaptive networks can better model context from other modalities than existing methods, many of which employ massive transformer-based networks.1 1 Introduction Multimodal deep learning is an active field of re- search, where for a single event, one has infor- mation across multiple modalities, such as video, speech, and text. Human brains can easily and per- petually perceive the context of an event from such heterogeneous data; however, it is not a trivial task for a computer system. In order for the machine to gain a contextual understanding, heterogeneous inputs must be combined first. Combining, or more precisely, fusing multimodal inputs is, thus, a vital step for any multimodal task. Naturally, a better fusion method will help a multimodal system learn better, ultimately enhancing its performance for a given task. 1Code for our experiments: https://github.com/ demfier/philo/ The most common fusion technique used in the literature involves the concatenation of representa- tions from all the available modalities. However, this results in a shallow network (Ngiam et al., 2011), and the network focuses more on learning intra-modal features, ignoring inter-modal dynam- ics altogether. Later, Zadeh et al. (2017) proposed Tensor Fusion Network (TFN), which models the unimodal, bimodal, and trimodal interactions using a 3-fold Cartesian product. TFN performs better than simple concatenation; however, it imposes high computational requirements since it projects all the information from input modalities to a dense 3-D space as-is, without any prior information ex- traction. The computational overhead grows ex- ponentially with respect to the dimensionality of unimodal features. Liu et al. (2018) proposed a low-rank multimodal fusion technique (LMF) to ad- dress the previous problem. Such fusion techniques are useful but often result in a complex architecture. Moreover, the fusion methods mentioned above focus only on combining individual unimodal fea- tures rather than combining and extracting useful information simultaneously. This means that the final predictor module (decoder in a Seq2Seq net- work (Sutskever et al., 2014), for example) bears an additional responsibility of identifying useful signals to …

Content truncated to 500 words