Contrastive Video-Language Learning with Fine-grained Frame Sampling

Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 694–705 November 20–23, 2022. ©2022 Association for Computational Linguistics 694 Contrastive Video-Language Learning with Fine-grained Frame Sampling Zixu Wang1, Yujie Zhong2, Yishu Miao3, Lin Ma2, Lucia Specia1 1Language and Multimodal AI Lab (LAMA), Imperial College London 2Meituan Inc., 3Haiper.ai zixu.wang@imperial.ac.uk, jaszhong@hotmail.com, yishu.miao@haiper.ai forest.linma@gmail.com, l.specia@imperial.ac.uk Abstract Despite recent progress in video and language representation learning, the weak or sparse cor- respondence between the two modalities re- mains a bottleneck in the area. Most video- language models are trained via pair-level loss to predict whether a pair of video and text is aligned. However, even in paired video-text segments, only a subset of the frames are se- mantically relevant to the corresponding text, with the remainder representing noise; where the ratio of noisy frames is higher for longer videos. We propose FineCo (Fine-grained Contrastive Loss for Frame Sampling), an ap- proach to better learn video and language repre- sentations with a fine-grained contrastive objec- tive operating on video frames. It helps distil a video by selecting the frames that are seman- tically equivalent to the text, improving cross- modal correspondence. Building on the well es- tablished VideoCLIP model as a starting point, FineCo achieves state-of-the-art performance on YouCookII, a text-video retrieval benchmark with long videos. FineCo also achieves compet- itive results on text-video retrieval (MSR-VTT), and video question answering datasets (MSR- VTT QA and MSR-VTT MC) with shorter videos. 1 Introduction Human perception is multimodal, including visual, textual, and audial information. To achieve human- level perceptional ability, intelligent systems need to understand and interpret these multimodal sig- nals and summarise the relevant information in them. Learning from video and language data has received significant attention in recent multimodal machine learning work for downstream tasks that require joint understanding of video and textual in- formation, including text-video retrieval (Lin et al., 2014; Liu et al., 2019; Miech et al., 2018; Wang et al., 2016; Bain et al., 2021), video question an- swering (Fan et al., 2019; Yang et al., 2021; Huang et al., 2020; Jiang et al., 2020; Le et al., 2020; Lei flip the pancakes when the edge turns brown mince the tuna and add it to a bowl Figure 1: Illustration of the weak correspondence prob- lem in video-language learning. Given a pair of video and its text (e.g. caption, instruction, or transcription), only a subset of the frames (here indicated by coloured bounding boxes) is semantically aligned to the textual content. The remaining frames represent irrelevant vi- sual information and will not contribute to language grounding on videos. et al., 2021), and video captioning (Ging et al., 2020; Luo et al., 2020; Zhang et al., 2020b). In most of this work, contrastive learning (Gutmann and Hyvärinen, 2010) is used as training objective. The aim of a cross-modal contrastive loss is to maximise the similarity between an aligned video- text pair …

Content truncated to 500 words