VLM: Task-agnostic Video-Language Model Pre-training for Video Understanding

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4227–4239 August 1–6, 2021. ©2021 Association for Computational Linguistics 4227 VLM: Task-agnostic Video-Language Model Pre-training for Video Understanding Hu Xu1, Gargi Ghosh1, Po-Yao Huang12, Prahal Arora1, Masoumeh Aminzadeh1 Christoph Feichtenhofer1, Florian Metze1 and Luke Zettlemoyer1 1Facebook AI 2Carnegie Mellon University {huxu,gghosh,berniehuang,prarora,masoumeha, feichtenhofer,fmetze,lsz}@fb.com Abstract We present a simplified, task-agnostic multi- modal pre-training approach that can accept either video or text input, or both for a va- riety of end tasks. Existing pre-training are task-specific by adopting either a single cross- modal encoder that requires both modalities, limiting their use for retrieval-style end tasks or more complex multitask learning with two unimodal encoders, limiting early cross-modal fusion. We instead introduce new pretraining masking schemes that better mix across modal- ities (e.g. by forcing masks for text to pre- dict the closest video embeddings) while also maintaining separability (e.g. unimodal pre- dictions are sometimes required, without us- ing all the input). Experimental results show strong performance across a wider range of tasks than any previous methods, often outper- forming task-specific pre-training1. 1 Introduction We study the challenge of achieving task-agnostic pre-training for multimodal video understanding, building on recent unimodal approaches such as pretrained language models for text (Peters et al., 2018; Devlin et al., 2019). Although certain lan- guage models are near task-agnostic (Devlin et al., 2019; Lewis et al., 2020) on NLP tasks, being task- agnostic on multi-modal tasks are more challeng- ing due to cross-modal tasks such as text-video re- trieval. Existing video-and-language pre-trainings are task-specific, which adopt either (1) a cross- modal single encoder (Sun et al., 2019b,a; Zhu and Yang, 2020) favoring tasks that require cross-modal reasoning (e.g. video captioning), or (2) multiple unimodal encoders/decoders (Miech et al., 2019, 2020; Li et al., 2020b; Luo et al., 2020; Korbar et al., 2020) combining specific tasks that require separately embedding each modality (e.g. video 1Code will be released under fairseq. Figure 1: Existing models (upper figure) adopt com- plex architectures and multiple task-specific training to merge two streams of data to cover a wide range of downstream tasks (such as retrieval or text generation). Our video-language model (VLM) (lower figure) uses a single BERT encoder for task-agnostic pre-training (e.g. only masking tokens, no matching or alignment for specific end tasks) in a joint feature space, while still covering a wide range of tasks (see Figure 3). retrieval). We instead show that it is possible to pre- train a task-agnostic model called video-language model (VLM) that can accept text, video, or both as input. As shown in Figure 1, this task-agnostic single encoder approach has several advantages: (1) it reduces the complexity of pre-training with mul- tiple losses and models (e.g. Luo et al. (2020)), and (2) it holds less assumption on being close to end tasks as in retrieval-based pre-training Miech et al. (2020) and is as general as classic LMs, and (3) it encourages feature sharing among modalities when present, without sacrificing separability, and (4) it is more parameter …

Content truncated to 500 words