End-to-end Dense Video Captioning as Sequence Generation

Proceedings of the 29th International Conference on Computational Linguistics, pages 5651–5665 October 12–17, 2022. 5651 End-to-end Dense Video Captioning as Sequence Generation Wanrong Zhu1, Bo Pang2, Ashish V. Thapliyal2, William Yang Wang1, Radu Soricut2 1UC Santa Barbara, 2Google Research {wanrongzhu,william}@cs.ucsb.edu {bopang,asht,rsoricut}@google.com Abstract Dense video captioning aims to identify the events of interest in an input video, and gen- erate descriptive captions for each event. Pre- vious approaches usually follow a two-stage generative process, which first proposes a seg- ment for each event, then renders a caption for each identified segment. Recent advances in large-scale sequence generation pretraining have seen great success in unifying task for- mulation for a great variety of tasks, but so far, more complex tasks such as dense video captioning are not able to fully utilize this pow- erful paradigm. In this work, we show how to model the two subtasks of dense video cap- tioning jointly as one sequence generation task, and simultaneously predict the events and the corresponding descriptions. Experiments on YouCook2 and ViTT show encouraging results and indicate the feasibility of training complex tasks such as end-to-end dense video caption- ing integrated into large-scale pretrained mod- els. 1 Introduction Online videos have become an important source of knowledge and skills (O’Neil-Hart, 2017). In order to help users locate information of interest, search engines and video platforms often show anchors at “key moments”, usually accompanied by descrip- tions of the segment’s content (Baheti, 2019). This is a direct application of the dense video caption- ing task (Krishna et al., 2017), thus methods for improving performance on this task are relevant to any video platform. Intuitively, dense video captioning can be de- composed into two subtasks: event localization and segment-level video captioning. Following this approach, prior work (Krishna et al., 2017; Zhou et al., 2018a; Li et al., 2018; Wang et al., 2018; Zhou et al., 2018c; Mun et al., 2019; Iashin and Rahtu, 2020) has used a two-stage, “localize-then- describe” pipeline. Such methods usually involve two separate modules with different underlying model architectures for event localization and event captioning, with captions for dense events rendered based on the predicted event spans. Recently, with the advance of large-scale datasets and model architectures, there has been an explosion of pretrained multimodal (for text, im- age, video) Transformer models (Tan and Bansal, 2019; Sun et al., 2019; Li et al., 2019; Luo et al., 2020; Li et al., 2020a,b; Gan et al., 2020; Kim et al., 2021). Such models have proved to be highly effective when fine-tuned for a wide range of downstream tasks, such as visual question an- swering (Agrawal et al., 2015), image caption- ing (Chen et al., 2015), visual common sense rea- soning (Zellers et al., 2019), visual entailment (Xie et al., 2019), etc. These end-tasks can be expressed as sequence generation tasks in a straightforward manner. In contrast, this is non-trivial for dense video captioning, as the segmentation subtask does not lend itself naturally to such a formulation. Does this mean more complex tasks …

Content truncated to 500 words