Modality-specific Learning Rates for Effective Multimodal Additive Late-fusion

Findings of the Association for Computational Linguistics: ACL 2022, pages 1824 - 1834 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Modality-specific Learning Rates for Effective Multimodal Additive Late-fusion Yiqun Yao Computer Science and Engineering University of Michigan yaoyq@umich.edu Rada Mihalcea Computer Science and Engineering University of Michigan mihalcea@umich.edu Abstract In multimodal machine learning, additive late- fusion is a straightforward approach to com- bine the feature representations from different modalities, in which the final prediction can be formulated as the sum of unimodal predictions. While it has been found that certain late-fusion models can achieve competitive performance with lower computational costs compared to complex multimodal interactive models, how to effectively search for a good late-fusion model is still an open question. Moreover, for dif- ferent modalities, the best unimodal models may work under significantly different learn- ing rates due to the nature of the modality and the computational flow of the model; thus, se- lecting a global learning rate for late-fusion models can result in a vanishing gradient for some modalities. To help address these issues, we propose a Modality-Specific Learning Rate (MSLR) method to effectively build late-fusion multimodal models from fine-tuned unimodal models. We investigate three different strate- gies to assign learning rates to different modali- ties. Our experiments show that MSLR outper- forms global learning rates on multiple tasks and settings, and enables the models to effec- tively learn each modality. 1 Introduction Multimodal machine learning aims to jointly under- stand and process the inputs from different modal- ities (e.g., language, audio, vision). This usually requires a model to have the ability to incorpo- rate the feature representations from each modality into a joint representation (the “multimodal fusion” problem). There are two types of commonly-used multimodal fusion methods: late-fusion and multi- modal interaction. Late-fusion methods rely on the representation vectors computed from unimodal encoders, which are then combined into a joint representation using operations such as addition, multiplication (Kim et al., 2016), bi-linear pool- ing (Fukui et al., 2016; Yu et al., 2017b), and so on. Multimodal interactive methods apply com- plex operations such as cross-modal attention (Yu et al., 2017a), modulation (Yao et al., 2018), and multi-head self-attention such as multimodal trans- formers (Tan and Bansal, 2019; Tsai et al., 2019). Despite the intuition that multimodal interac- tion leverages the inter-dependency across differ- ent modalities, (Hessel and Lee, 2020) proposed that there is a method to simulate the outputs of an additive late-fusion model that has the closest possible performance to an arbitrary interactive model (but not how to find the specific structure). According to the experimental results in (Hessel and Lee, 2020), the accuracy of the closest addi- tive models is competitive with the corresponding interactive models in some selected tasks. This indicates that: (1) Currently, some interactive mod- els are not strong enough to catch the complex real-world inter-dependencies between modalities. Studying the upper-bound of late-fusion methods can help evaluate the limitations of interactive mod- els. (2) The application of late-fusion models is still open …

Content truncated to 500 words