Bridge Video and Text with Cascade Syntactic Structure

Proceedings of the 27th International Conference on Computational Linguistics, pages 3576–3585 Santa Fe, New Mexico, USA, August 20-26, 2018. 3576 Bridge Video and Text with Cascade Syntactic Structure Guolong Wang, Zheng Qin, Kaiping Xu, Kai Huang and Shuxiong Ye Tsinghua University wanggl16@mails.tsinghua.edu.cn, qingzh@tsinghua.edu.cn {xkp13, huang-k15, ysx15}@mails.tsinghua.edu.cn Abstract We present a video captioning approach that encodes features by progressively completing syn- tactic structure (LSTM-CSS). To construct basic syntactic structure (i.e., subject, predicate, and object), we use a Conditional Random Field to label semantic representations (i.e., motions, ob- jects). We argue that in order to improve the comprehensiveness of the description, the local features within object regions can be used to generate complementary syntactic elements (e.g., attribute, adverbial). Inspired by redundancy of human receptors, we utilize a Region Proposal Network to focus on the object regions. To model the final temporal dynamics, Recurrent Neural Network with Path Embeddings is adopted. We demonstrate the effectiveness of LSTM-CSS on generating natural sentences: 42.3% and 28.5% in terms of BLEU@4 and METEOR. Superior performance when compared to state-of-the-art methods are reported on a large video description dataset (i.e., MSR-VTT-2016). 1 Introduction Video has become a ubiquitous way of communication on the Internet, podcast channels, as well as mo- bile devices. Accelerated by the explosive spread of video data, automatic analysis of semantic video content remains a promising area. The advances of this task can provide comparatively favorable precon- ditions for subsequent tasks, such as video retrieval, human-perception analysis, keyframe recommenda- tion(Chen et al., 2017). To encapsulate the informative dynamics in the video, researchers have started focusing on recognizing videos based on the predefined templates, such as (Kojima et al., 2002; Guadar- rama et al., 2014). Rohrbach et al. (2013) learned a CRF to assemble between different components of the input video and generated descriptions for videos. Xu et al. (2015) utilized a unified framework that jointly models video and the corresponding text sentences. Another line of work uses Recurrent Neural Network. In light of its success, the representations (e.g. key objects, locations, motions, and scenes) can be concatenated into an input sequence and then trans- lated to a natural sentence (Donahue et al., 2015). Follow-up works investigate the modeling of not only video contents and their spatio-temporal relationships, but also the syntactical structure (Venugopalan et al., 2015). More recently, a visual-semantic embedding learning technique has been proposed to model video content and textual semantics as a regularizer in Long Short-Term Memory architecture (Pan et al., 2016). This was extended by (Hori et al., 2017) where they utilized a modality-dependent attention mechanism. Over the past years, researchers have studied multiple strategies to effectively bridge the visual con- tent and textual description based on some fresh ideas. (Pan et al., 2017) incorporated the transferred semantic attributes learned from images and videos into the CNN plus RNN framework. (Baraldi et al., 2017) proposed a novel LSTM cell which can modify the temporal connections of the encoding layer according to the identified discontinuity points among frames. (Kaufman et al., …

Content truncated to 500 words