Modality Alignment between Deep Representations for Effective Video-and-Language Learning

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 2759–2770 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 2759 Modality Alignment between Deep Representations for Effective Video-and-Language Learning Hyeongu Yun12*, Yongil Kim12*, Kyomin Jung12† 1Automation and Systems Research Institute, 2Seoul National University {youaredead,miles94,kjung}@snu.ac.kr Abstract Video-and-Language learning, such as video question answering or video captioning, is the next challenge in the deep learning society, as it pursues the way how human intelligence perceives everyday life. These tasks require the ability of multi-modal reasoning which is to handle both visual information and text information simultaneously across time. In this point of view, a cross-modality attention module that fuses video representation and text representation takes a critical role in most recent approaches. However, existing Video-and-Language models merely compute the attention weights without considering the different characteristics of video modality and text modality. Such na¨ıve attention module hinders the current models to fully enjoy the strength of cross-modality. In this paper, we propose a novel Modality Alignment method that benefits the cross-modality attention module by guiding it to easily amalgamate multiple modalities. Specifically, we exploit Centered Kernel Alignment (CKA) which was originally proposed to measure the similarity between two deep representations. Our method directly optimizes CKA to make an alignment between video and text embedding representations, hence it aids the cross-modality attention module to combine information over different modalities. Experiments on real-world Video QA tasks demonstrate that our method outperforms conventional multi-modal methods significantly with +3.57% accuracy increment compared to the baseline in a popular benchmark dataset. Additionally, in a synthetic data environment, we show that learning the alignment with our method boosts the performance of the cross-modality attention. Keywords: Multi-modality, Video-and-Language Learning, Cross-modality Attention 1. Introduction For deep learning researchers, multi-modality recently became an important keyword as multi-modal mod- els have shown the ability to collate plentiful informa- tion scattered over various modalities (Cadene et al., 2019; Li et al., 2020b; Shen et al., 2018). In particu- lar, Video-and-Language learning which includes both video modality and text modality is attracting a huge attention (Lei et al., 2020a; Zellers et al., 2021; Li et al., 2020a; Miech et al., 2019; Yu et al., 2019; Yang et al., 2021). Specifically, Video-and-Language learn- ing such as video captioning or video question answer- ing require the ability of reasoning over both time and multiple modality. For example, a video question an- swering model should be able to find appropriate vi- sual information in a video frame sequence with a given question. That is to say, capturing the relationship be- tween video information and text information is impor- tant for a multi-modal model for Video-and-Language learning. Cross modality attention module which combines cor- relation over different modalities becomes a critical component for Video-and-Language learning (Ye et al., 2019; Lu et al., 2019; Chen et al., 2020b).Generally, the attention mechanism induces a model to learn the most important representation among the whole se- quence with a given query. For single …

Content truncated to 500 words