HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 2046–2065, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 2046 HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training Linjie Li∗, Yen-Chun Chen∗, Yu Cheng, Zhe Gan, Licheng Yu, Jingjing Liu Microsoft Dynamics 365 AI Research {lindsey.li, yen-chun.chen, yu.cheng, zhe.gan, licheng.yu, jingjl}@microsoft.com Abstract We present HERO, a novel framework for large-scale video+language omni- representation learning. HERO encodes multimodal inputs in a hierarchical structure, where local context of a video frame is captured by a Cross-modal Transformer via multimodal fusion, and global video context is captured by a Temporal Transformer. In addi- tion to standard Masked Language Modeling (MLM) and Masked Frame Modeling (MFM) objectives, we design two new pre-training tasks: (i) Video-Subtitle Matching (VSM), where the model predicts both global and local temporal alignment; and (ii) Frame Order Modeling (FOM), where the model predicts the right order of shuffled video frames. HERO is jointly trained on HowTo100M and large-scale TV datasets to gain deep under- standing of complex social dynamics with multi-character interactions. Comprehensive experiments demonstrate that HERO achieves new state of the art on multiple benchmarks over Text-based Video/Video-moment Re- trieval, Video Question Answering (QA), Video-and-language Inference and Video Captioning tasks across different domains. We also introduce two new challenging benchmarks How2QA and How2R for Video QA and Retrieval, collected from diverse video content over multimodalities.1 1 Introduction Inspired by BERT (Devlin et al., 2019), large- scale multimodal pre-training has prevailed in the realm of vision-and-language research (Lu et al., 2019; Tan and Bansal, 2019; Chen et al., 2020b). There are many early players in the area, including ViLBERT (Lu et al., 2019), LXMERT (Tan and ∗ Equal contribution. 1Code and new datasets publicly available at: https: //github.com/linjieli222/HERO. Bansal, 2019), UNITER (Chen et al., 2020b), VL- BERT (Su et al., 2020) and Unicoder-VL (Li et al., 2020a). However, most large-scale pre-trained models are tailored for static images, not dynamic videos. VideoBERT (Sun et al., 2019b) is the first to apply BERT to learn joint embedding for video- text pairs. But since only discrete tokens are used to represent video frames, rich video frame features are not fully utilized. To remedy this, CBT (Sun et al., 2019a) proposes to use a contrastive loss, but mainly for video representation learning alone, with text input only considered as side information. UniViLM (Luo et al., 2020) takes a step further and considers both understanding and generation tasks. Several constraints inherently limit the success of existing models. (i) Most model designs are di- rect adaptation of BERT, taking simple concatena- tion of subtitle sentences and visual frames as input, while losing the temporal alignment between video and text modalities. (ii) Pre-training tasks are di- rectly borrowed from image+text pre-training meth- ods, without exploiting the sequential nature of videos. (iii) Compared to diverse image domains investigated in existing work, video datasets used in current models are restricted to cooking or narrated instructional videos (Miech et al., 2019), exclud- ing video sources that contain dynamic …

Content truncated to 500 words