Multimodal Pretraining for Dense Video Captioning

Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 470–490 December 4 - 7, 2020. c⃝2020 Association for Computational Linguistics 470 Multimodal Pretraining for Dense Video Captioning Gabriel Huang1∗, Bo Pang2, Zhenhai Zhu2, Clara Rivera2, Radu Soricut2 1Mila & University of Montreal 2Google Research gabriel.huang@umontreal.ca {bopang,zhenhai,rivera,rsoricut}@google.com Abstract Learning specific hands-on skills such as cook- ing, car maintenance, and home repairs in- creasingly happens via instructional videos. The user experience with such videos is known to be improved by meta-information such as time-stamped annotations for the main steps involved. Generating such annotations au- tomatically is challenging, and we describe here two relevant contributions. First, we construct and release a new dense video cap- tioning dataset, Video Timeline Tags (ViTT), featuring a variety of instructional videos to- gether with time-stamped annotations. Sec- ond, we explore several multimodal sequence- to-sequence pretraining strategies that lever- age large unsupervised datasets of videos and caption-like texts. We pretrain and subse- quently finetune dense video captioning mod- els using both YouCook2 and ViTT. We show that such models generalize well and are robust over a wide variety of instructional videos. 1 Introduction YouTube recently reported that a billion hours of videos were being watched on the platform every day (YouTubeBlog, 2017). In addition, the amount of time people spent watching online videos was estimated to grow at an average rate of 32% a year between 2013 and 2018, with an average person forecasted to watch 100 minutes of online videos per day in 2021 (ZenithMedia, 2019). An important reason for this fast-growing video consumption is information-seeking. For instance, people turn to YouTube “hungry for how-to and learning content” (O’Neil-Hart, 2018). Indeed, compared to traditional content format such as text, video carries richer information to satisfy such ∗This work was done while Gabriel Huang was an intern at Google Research. Groundtruth Varying stiching speeds Ø-Pretraining Showing other parts MASS-Pretraining Explaining how to do a stitch Figure 1: Dense video captioning using ViTT–trained models. For the given video scene, we show the ViTT annotation (Groundtruth) and model outputs (no pre- training and MASS-based pretraining). needs. But as a content media, videos are also in- herently more difficult to skim through, making it harder to quickly target the relevant part(s) of a video. Recognizing this difficulty, search engines started showing links to “key moments” within videos in search results, based on timestamps and short descriptions provided by the content creators themselves.1 This enables users to get a quick sense of what the video covers, and also to jump to a particular time in the video if so desired. This effort echoes prior work in the literature show- ing how users of instructional videos can benefit from human-curated meta-data, such as a timeline pointing to the successive steps of a tutorial (Kim et al., 2014; Margulieux et al., 2012; Weir et al., 2015). Producing such meta-data in an automatic way would greatly scale up …

Content truncated to 500 words