DeCEMBERT: Learning from Noisy Instructional Videos via Dense Captions and Entropy Minimization

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2415–2426 June 6–11, 2021. ©2021 Association for Computational Linguistics 2415 DECEMBERT: Learning from Noisy Instructional Videos via Dense Captions and Entropy Minimization Zineng Tang Jie Lei Mohit Bansal Department of Computer Science University of North Carolina at Chapel Hill {terran, jielei, mbansal}@cs.unc.edu Abstract Leveraging large-scale unlabeled web videos such as instructional videos for pre-training followed by task-specific finetuning has be- come the de facto approach for many video- and-language tasks. However, these instruc- tional videos are very noisy, the accompanying ASR narrations are often incomplete, and can be irrelevant to or temporally misaligned with the visual content, limiting the performance of the models trained on such data. To ad- dress these issues, we propose an improved video-and-language pre-training method that first adds automatically-extracted dense region captions from the video frames as auxiliary text input, to provide informative visual cues for learning better video and language associ- ations. Second, to alleviate the temporal mis- alignment issue, our method incorporates an entropy minimization-based constrained atten- tion loss, to encourage the model to automati- cally focus on the correct caption from a pool of candidate ASR captions. Our overall ap- proach is named DECEMBERT (Dense Cap- tions and Entropy Minimization). Comprehen- sive experiments on three video-and-language tasks (text-to-video retrieval, video caption- ing, and video question answering) across five datasets demonstrate that our approach outper- forms previous state-of-the-art methods. Ab- lation studies on pre-training and downstream tasks show that adding dense captions and con- strained attention loss help improve the model performance. Lastly, we also provide attention visualization to show the effect of applying the proposed constrained attention loss.1 1 Introduction Video and language are ubiquitous in the world we live. The ability to understand the interplay of video and language is thus essential for intelligent agents to operate in real-world scenario. Past suc- cess in video-and-language has mostly been driven 1Code and models: https://github.com/ zinengtang/DeCEMBERT by supervised learning, where models are learned on manually labeled data for a particular task (e.g., text-to-video retrieval). However, manually anno- tating video and language data is very expensive, hence limiting the scale of such datasets, and con- sequently also limiting the performance of models trained on the datasets. The self-supervised pre- training then finetuning paradigm offers an easy and generic solution to this dilemma, where models are first pre-trained on large-scale unlabeled data by performing various “proxy tasks”, followed by finetuning the pre-trained model on downstream tasks where data is often limited. Recent advances on language pre-training (De- vlin et al., 2019; Liu et al., 2019) demonstrate the effectiveness of this approach, where transformer- based (Vaswani et al., 2017) models pre-trained on large-scale unlabeled text corpus has shown to perform remarkably well across a wide range of natural language tasks (Rajpurkar et al., 2016; Williams et al., 2017; Zellers et al., 2018; Wang et al., 2018). Following this momentum, mul- timodal pre-training (Tan and Bansal, 2019; Lu …

Content truncated to 500 words