Unimodal and Crossmodal Refinement Network for Multimodal Sequence Fusion

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9143–9153 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 9143 Unimodal and Crossmodal Refinement Network for Multimodal Sequence Fusion Xiaobao Guo1, 2, Adams Kong1, Huan Zhou3, Xianfeng Wang3, and Min Wang3 1 School of Computer Science and Engineering, Nanyang Technological University 2 Rapid-Rich Object Search (ROSE) Lab, IGP, Nanyang Technological University 3 AI Application Research Center (AARC), Huawei Technologies xiaobao001@e.ntu.edu.sg, adamskong@ntu.edu.sg {zhou.huan,wangxianfeng10,wangmin5}@huawei.com Abstract Effective unimodal representation and com- plementary crossmodal representation fusion are both important in multimodal representa- tion learning. Prior works often modulate one modal feature to another straightforwardly and thus, underutilizing both unimodal and crossmodal representation refinements, which incurs a bottleneck of performance improve- ment. In this paper, Unimodal and Crossmodal Refinement Network (UCRN) is proposed to enhance both unimodal and crossmodal rep- resentations. Specifically, to improve uni- modal representations, a unimodal refinement module is designed to refine modality-specific learning via iteratively updating the distribu- tion with transformer-based attention layers. Self-quality improvement layers are followed to generate the desired weighted representa- tions progressively. Subsequently, those uni- modal representations are projected into a common latent space, regularized by a mul- timodal Jensen-Shannon divergence loss for better crossmodal refinement. Lastly, a cross- modal refinement module is employed to inte- grate all information. By hierarchical explo- rations on unimodal, bimodal, and trimodal interactions, UCRN is highly robust against missing modality and noisy data. Experimen- tal results on MOSI and MOSEI datasets illus- trated that the proposed UCRN outperforms re- cent state-of-the-art techniques and its robust- ness is highly preferred in real multimodal se- quence fusion scenarios. Codes will be shared publicly1. 1 Introduction Motivated by recent research achievements on modality representations in language (Pennington et al., 2014; Devlin et al., 2019; Brown et al., 2020), audio (Degottex et al., 2014; Chen et al., 2018; Li et al., 2019), and vision (He et al., 2016; Woo 1https://github.com/HW-AARC-CLUB/ emnlp2021_ucrn et al., 2018; Li and Deng, 2020), multimodal learn- ing (Baltrušaitis et al., 2018) that aims to efficiently extract joint representations from multiple sensory data, has drawn much attention recently. By mining complementary relations across modalities, multi- modal fusion could present a more reliable and comprehensive interpretation of the world. With the development of various network archi- tectures, considerable progress has been achieved in multimodal fusion (Williams et al., 2018; Tsai et al., 2019; Mai et al., 2020; Zadeh et al., 2019), showing that multimodal representation outper- forms unimodal ones on emotion and sentiment prediction tasks. Additionally, recent works seek to improve the efficacy of the multimodal fusion methods by assuming that one modality can be translated to another (Tsai et al., 2019; Mai et al., 2020), or modulated by pivot modality (Delbrouck et al., 2020), so as to align pairwise representations in a common space. However, converting one modality to another modality appears to be inadequate in projecting all modalities into one feature space. A specific pattern may not coexist in all modalities (e.g., it ex- presses happiness in language/audio …

Content truncated to 500 words