LiteVL: Efficient Video-Language Learning with Enhanced Spatial-Temporal Modeling

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7985 - 7997 December 7-11, 2022 ©2022 Association for Computational Linguistics LiteVL: Efficient Video-Language Learning with Enhanced Spatial-Temporal Modeling Dongsheng Chen1, Chaofan Tao2, Lu Hou3, Lifeng Shang3, Xin Jiang3, Qun Liu3 1Peking University 2The University of Hong Kong 3Huawei Noah’s Ark Lab chends@stu.pku.edu.cn, cftao@connect.hku.hk {houlu3,shang.lifeng,jiang.xin,qun.liu}@huawei.com Abstract Recent large-scale video-language pre-trained models have shown appealing performance on various downstream tasks. However, the pre- training process is computationally expensive due to the requirement of millions of video- text pairs and the redundant data structure of each video. To mitigate these problems, we propose LiteVL, which adapts a pre-trained image-language model BLIP into a video-text model directly on downstream tasks, without heavy pre-training. To enhance the temporal modeling lacking in the image-language model, we propose to add temporal attention modules in the image encoder of BLIP with dynamic temporal scaling. Besides the model-wise adaptation, we also propose a non-parametric pooling mechanism to adaptively reweight the fine-grained video embedding conditioned on the text. Experimental results on text-video retrieval and video question answering show that the proposed LiteVL even outperforms previous video-language pre-trained models by a clear margin, though without any video- language pre-training. 1 Introduction The increasing popularity of videos in various so- cial media has aroused the interest in efficient mod- eling of videos and their connections with other modalities like texts. Video-language modeling targets at learning a shared multimodal semantic space for videos and texts to facilitate downstream tasks like text-video retrieval and video question an- swering (VideoQA). Previous video-language mod- eling usually relies on pre-training on a large-scale video-text pair, via video-text contrastive learning (Luo et al., 2021; Li et al., 2022a; Gorti et al., 2022), video-text matching (Fan et al., 2019; Luo et al., 2020; Li et al., 2022a), masked language modeling (Devlin et al., 2019) and masked frame modeling (Sun et al., 2019); or extra object detec- tors to extract fine-grained visual features (Zhu and Yang, 2020; Chen et al., 2020). However, both pre- training on video-text pairs and using an off-the- shelf detector are computationally expensive and inefficient. Inaccurate detection results on limited categories may also lead to inferior performance. In the unimodal video domain, TimeSformer (Bertasius et al., 2021) fails to pre-train video en- coder directly on large-scale video datasets, but obtains good performance by initializing from a pre-trained Transformer-based ViT (Dosovitskiy et al., 2021) image encoder and training additional temporal attention modules directly on downstream video tasks. Similarly, ViViT (Arnab et al., 2021) also takes advantage of the already well-learned spatial visual representation in a pre-trained ViT model, and effectively adapts it for video tasks by directly fine-tuning on comparatively small down- stream video datasets. Inspired by TimeSformer and ViViT, in this pa- per, we also consider extending an image-language pre-trained model for video-text tasks without pre- training on video-text pairs. This requires us to not only leverage the already-learned alignment between spatial visual information and text in the image-language models, but also …

Content truncated to 500 words