MM-Align: Learning Optimal Transport-based Alignment Dynamics for Fast and Accurate Inference on Missing Modality Sequences

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10498 - 10511 December 7-11, 2022 ©2022 Association for Computational Linguistics MM-Align: Learning Optimal Transport-based Alignment Dynamics for Fast and Accurate Inference on Missing Modality Sequences Wei Han DeCLaRe Hui Chen DeCLaRe Min-Yen Kan♣ Soujanya Poria DeCLaRe DeCLaReDeCLaRelab, Singapore University of Technology and Design, Singapore ♣ National University of Singapore, Singapore {wei_han,hui_chen}@mymail.sutd.edu.sg kanmy@comp.nus.edu.sg,sporia@sutd.edu.sg Abstract Existing multimodal tasks mostly target at the complete input modality setting, i.e., each modality is either complete or completely miss- ing in both training and test sets. However, the randomly missing situations have still been underexplored. In this paper, we present a novel approach named MM-Align to address the missing-modality inference problem. Con- cretely, we propose 1) an alignment dynam- ics learning module based on the theory of optimal transport (OT) for indirect missing data imputation; 2) a denoising training algo- rithm to simultaneously enhance the imputa- tion results and backbone network performance. Compared with previous methods which de- vote to reconstructing the missing inputs, MM- Align learns to capture and imitate the align- ment dynamics between modality sequences. Results of comprehensive experiments on three datasets covering two multimodal tasks empiri- cally demonstrate that our method can perform more accurate and faster inference and relieve overfitting under various missing conditions. Our code is available at https://github. com/declare-lab/MM-Align. 1 Introduction The topic of multimodal learning has grown un- precedentedly prevalent in recent years (Ramachan- dram and Taylor, 2017; Baltrušaitis et al., 2018), ranging from a variety of machine learning tasks such as computer vision (Zhu et al., 2017; Nam et al., 2017), natural langauge processing (Fei et al., 2021; Ilharco et al., 2021), autonomous driv- ing (Caesar et al., 2020) and medical care (Nascita et al., 2021), etc. Despite the promising achieve- ments in these fields, most of existent approaches assume a complete input modality setting of train- ing data, in which every modality is either complete or completely missing (at inference time) in both training and test sets (Pham et al., 2019; Tang et al., 2021; Zhao et al., 2021), as shown in Fig. 1a and 1b. Such synergies between train and test sets in the modality input patterns are usually far from the realistic scenario where there is a certain portion of data without parallel modality sequences, prob- ably due to noise pollution during collecting and preprocessing time. In other words, data from each modality are more probable to be missing at random (Fig.1c and 1d) than completely present or missing (Fig.1a and 1b) (Pham et al., 2019; Tang et al., 2021; Zhao et al., 2021). Based on the complete input modality setting, a family of popular routines regarding the missing-modality inference is to design intricate generative modules attached to the main network and train the model under full supervision with complete modality data. By minimizing a customized reconstruction loss, the data restoration (a.k.a. missing data imputa- tion (Van Buuren, 2018)) capability of the genera- tive modules is enhanced (Pham et al., …

Content truncated to 500 words