Multi-Task Video Captioning with Video and Entailment Generation

Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1273–1283 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1117 Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1273–1283 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1117 Multi-Task Video Captioning with Video and Entailment Generation Ramakanth Pasunuru and Mohit Bansal UNC Chapel Hill {ram, mbansal}@cs.unc.edu Abstract Video captioning, the task of describing the content of a video, has seen some promising improvements in recent years with sequence-to-sequence models, but accurately learning the temporal and log- ical dynamics involved in the task still re- mains a challenge, especially given the lack of sufficient annotated data. We im- prove video captioning by sharing knowl- edge with two related directed-generation tasks: a temporally-directed unsuper- vised video prediction task to learn richer context-aware video encoder representa- tions, and a logically-directed language entailment generation task to learn bet- ter video-entailing caption decoder rep- resentations. For this, we present a many-to-many multi-task learning model that shares parameters across the encoders and decoders of the three tasks. We achieve significant improvements and the new state-of-the-art on several standard video captioning datasets using diverse au- tomatic and human evaluations. We also show mutual multi-task improvements on the entailment generation task. 1 Introduction Video captioning is the task of automatically gen- erating a natural language description of the con- tent of a video, as shown in Fig. 1. It has various applications such as assistance to a visually im- paired person and improving the quality of online video search or retrieval. This task has gained re- cent momentum in the natural language process- ing and computer vision communities, esp. with the advent of powerful image processing features as well as sequence-to-sequence LSTM models. It Figure 1: A video captioning example from the YouTube2Text dataset, with the ground truth captions and our many-to-many multi-task model’s predicted caption. is also a step forward from static image captioning, because in addition to modeling the spatial visual features, the model also needs to learn the tempo- ral across-frame action dynamics and the logical storyline language dynamics. Previous work in video captioning (Venu- gopalan et al., 2015a; Pan et al., 2016b) has shown that recurrent neural networks (RNNs) are a good choice for modeling the temporal information in the video. A sequence-to-sequence model is then used to ‘translate’ the video to a caption. Venu- gopalan et al. (2016) showed linguistic improve- ments over this by fusing the decoder with external language models. Furthermore, an attention mech- anism between the video frames and the caption words captures some of the temporal matching re- lations better (Yao et al., 2015; Pan et al., 2016a). More recently, hierarchical two-level RNNs were proposed to allow for longer inputs and to model the full paragraph caption dynamics of long video clips (Pan et al., 2016a; Yu et al., 2016). Despite these recent improvements, video cap- tioning models still suffer from …

Content truncated to 500 words