Video Captioning with Multi-Faceted Attention

Video Captioning with Multi-Faceted Attention Xiang Long Tsinghua University longx13@mails.tsinghua.edu.cn Chuang Gan Tsinghua University ganchuang1990@gmail.com Gerard de Melo Rutgers University gdm@demelo.org Abstract Video captioning has attracted an increasing amount of interest, due in part to its potential for improved accessibility and information re- trieval. While existing methods rely on differ- ent kinds of visual features and model archi- tectures, they do not make full use of pertinent semantic cues. We present a unified and exten- sible framework to jointly leverage multiple sorts of visual features and semantic attributes. Our novel architecture builds on LSTMs with two multi-faceted attention layers. These first learn to automatically select the most salient visual features or semantic attributes, and then yield overall representations for the input and output of the sentence generation component via custom feature scaling operations. Exper- imental results on the challenging MSVD and MSR-VTT datasets show that our framework outperforms previous work and performs ro- bustly even in the presence of added noise to the features and attributes. 1 Introduction The task of automatically generating captions for videos has been receiving an increasing amount of attention. On YouTube, for example, every single minute, hundreds of hours of video content are up- loaded. Obviously, there is no way a person could sit and binge-watch these overwhelming amounts of video, so new techniques to search and quickly un- derstand them are highly sought. Generating cap- tions, i.e., short natural language descriptions, for videos is an important technique to address this chal- Video: 301 frames a monkey is playing with a dog. Semantic Attributes: “Monkey”, “Animal”, “Dog”, “Playing”, “Pulling”, “Grass”... Temporal Features: Resnet-152 Motion Features: C3D × 18 4096 × 301 2048 Figure 1: Example video with extracted visual features, semantic attributes, and the generated caption as output. lenge, while also greatly improving their accessibil- ity for blind and visually impaired users. The study of video captioning has an extensive history but remains challenging, given the difficul- ties of video interpretation, natural language genera- tion, and their interplay. Video interpretation hinges on our ability to make sense of a stream of video frames and of the relationships between consecutive frames. The generated output needs to be a gram- matically correct sequence of words, and different parts of the output caption may pertain to different parts of the video. In previous work, 3D ConvNets (Du et al., 2015) have been proposed to capture motion information in short videos, while LSTMs (Hochreiter and Schmidhuber, 1997) can be used to generate natural language, and a variety of different visual attention models (Yao et al., 2015; Pan et al., 2016a; Yu et al., 2016; Long et al., 2018) have been deployed, attempting to capture the relationship be- tween caption words and the video content. These methods, however, only make use of visual 173 Transactions of the Association for Computational Linguistics, vol. 6, pp. 173–184, 2018. Action Editor: Jianfeng Gao. Submission batch: 9/2017; Revision batch: 11/2017; Published 3/2018. c⃝2018 Association for Computational Linguistics. Distributed under a CC-BY 4.0 license. information …

Content truncated to 500 words