Incorporating Semantic Attention in Video Description Generation

Incorporating Semantic Attention in Video Description Generation Natsuda Laokulrat1, Naoaki Okazaki1,2, Hideki Nakayama1,3 1Artificial Intelligence Research Center (AIRC), AIST, Japan 2Tokyo Institute of Technology, Japan 3The University of Tokyo, Japan natsudalkr@gmail.com, okazaki@c.titech.ac.jp, nakayama@ci.i.u-tokyo.ac.jp Abstract Automatically generating video description is one of the approaches to enable computers to deeply understand videos, which can have a great impact and can be useful to many other applications. However, generated descriptions by computers often fail to correctly mention objects and actions appearing in the videos. This work aims to alleviate this problem by including external fine-grained visual information, which can be detected from all video frames, in the description generation model. In this paper, we propose an LSTM-based sequence-to-sequence model with semantic attention mechanism for video description generation. The model is flexible so that we can change the source of the external information without affecting the encoding and decoding parts of the model. The results show that using semantic attention to selectively focus on external fine-grained visual information can guide the system to correctly mention objects and actions in videos and have a better quality of video descriptions. Keywords: video description generation, deep learning, RNN, attention 1. Introduction In the past few years, the image captioning task has been gaining popularity among researchers and high-quality im- age captions can be generated by deep learning techniques (Vinyals et al., 2014; Karpathy and Fei-Fei, 2015; Fang et al., 2015; Xu et al., 2015; You et al., 2016). The first work in this field was proposed by Vinyals et al. (2014). They proposed an end-to-end system consisting of a Convolu- tional Neural Network (CNN) and a Recurrent Neural Net- work (RNN). The output of the last fully connected layer of the image classification CNN is used as an image feature and then is injected into the RNN-based language model to produce a meaningful sentence. Later, Xu et al. (2015) has proposed an attention-based framework for image caption- ing which can selectively focus on a portion of an image while producing each word. However, researchers still can- not achieve a satisfying quality of video descriptions gen- erated by computers yet. As with image captioning, auto- matic video description generation combines two fields of artificial intelligence, computer vision and natural language processing, and has also been tackled by the combination of RNN and CNN. Venugopalan et al. (2015b) proposed the first end-to-end system to translate a video to natural language by extend- ing the CNN-RNN encoder-decoder framework for image captioning proposed by Vinyals et al. (2014) to generate de- scriptions for videos. They performed a mean pooling over CNN feature vectors of frames to generate a single vector representation for a video, and then use the vector as input to the RNN decoder to generate a sentence, ignoring the temporal ordering of videos. Subsequently, they have pro- posed an RNN-based sequence-to-sequence model for gen- erating descriptions of videos (Venugopalan et al., 2015a). They used 2 layers of RNN for both encoding the videos and decoding to sentences, so their …

Content truncated to 500 words