VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6787–6800 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 6787 VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding Hu Xu1, Gargi Ghosh1, Po-Yao Huang12, Dmytro Okhonko1, Armen Aghajanyan1 Florian Metze,1 Luke Zettlemoyer1 and Christoph Feichtenhofer1 1Facebook AI 2Carnegie Mellon University {huxu,gghosh,berniehuang,oxo,armenag fmetze,lsz,feichtenhofer}@fb.com Abstract We present VideoCLIP, a contrastive ap- proach to pre-train a unified model for zero- shot video and text understanding, with- out using any labels on downstream tasks. VideoCLIP trains a transformer for video and text by contrasting temporally overlapping positive video-text pairs with hard negatives from nearest neighbor retrieval. Our exper- iments on a diverse series of downstream tasks, including sequence-level text-video re- trieval, VideoQA, token-level action localiza- tion, and action segmentation reveal state-of- the-art performance, surpassing prior work, and in some cases even outperforming su- pervised approaches. Code is made avail- able at https://github.com/pytorch/ fairseq/tree/main/examples/MMPT. 1 Introduction The popular “pre-training + fine-tuning” paradigm has revolutionized NLP (Devlin et al., 2019; Liu et al., 2019b; Yang et al., 2019; Lewis et al., 2020b) and CV (Chen et al., 2020a; He et al., 2020) over the last few years. Although models trained this way can achieve impressive performance, they still require task-specific annotated data and fine-tuning for each end task. Recent work adopt pre-training for zero-shot transfer to end tasks without fine- tuning, including GPT (Radford et al., 2018, 2019; Brown et al., 2020) for NLP tasks and CLIP (Rad- ford et al., 2021) for image classification. This paper focuses on pre-training for zero-shot transfer to video-text understanding tasks. Our ap- proach pre-trains a Transformer model (Vaswani et al., 2017; Devlin et al., 2019) with a contrastive objective (Oord et al., 2018; Chen et al., 2020a) us- ing pairs of video-text clips. Different from CLIP that scales pre-training data for zero-shot transfer to image classification on an explicitly assembled dataset using a simple contrastive objective (Chen et al., 2020a), this paper uses a publicly established Video 1 First, you need.. Next, pour sauce.. 1 Overlapping video-text clips 2 Retrieve harder videos Video K VideoCLIP: Contrastive learning with hard-retrieved negatives and overlapping positives for video-text pre-training. … time how to season a wok Video 2 now, wipe with a paper towel Retrieval Figure 1: VideoCLIP aims for zero-shot video under- standing via learning fine-grained association between video and text in a transformer using a contrastive ob- jective with two key novelties: (1) for positive pairs, we use video and text clips that are loosely temporarily overlapping instead of enforcing strict start/end times- tamp overlap; (2) for negative pairs, we employ a re- trieval based sampling technique that uses video clus- ters to form batches with mutually harder videos. pre-training dataset, HowTo100M (Miech et al., 2019), for zero-shot video understanding. We show that the resulting pre-trained model can be either directly applied to, or fine-tuned on, a series of video-text tasks at both the global sequence and local clip/token level. We find that straightforward objectives (Chen et al., …

Content truncated to 500 words