ExCL: Extractive Clip Localization Using Natural Language Descriptions

Proceedings of NAACL-HLT 2019, pages 1984–1990 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 1984 ExCL: Extractive Clip Localization Using Natural Language Descriptions Soham Ghosh1,∗ Anuva Agarwal1,∗ Zarana Parekh1,∗ Alexander Hauptmann1 1Language Technologies Institute Carnegie Mellon University {sohamg,anuvaa,zpp,alex}@cs.cmu.edu Abstract The task of retrieving clips within videos based on a given natural language query re- quires cross-modal reasoning over multiple frames. Prior approaches such as sliding win- dow classifiers are inefficient, while text-clip similarity driven ranking-based approaches such as segment proposal networks are far more complicated. In order to select the most relevant video clip corresponding to the given text description, we propose a novel extrac- tive approach that predicts the start and end frames by leveraging cross-modal interactions between the text and video - this removes the need to retrieve and re-rank multiple proposal segments. Using recurrent networks we en- code the two modalities into a joint representa- tion which is then used in different variants of start-end frame predictor networks. Through extensive experimentation and ablative analy- sis, we demonstrate that our simple and ele- gant approach significantly outperforms state of the art on two datasets and has comparable performance on a third. 1 Introduction Clip Localization is the task of selecting the rel- evant span of temporal frames in a video corre- sponding to a natural language description and has recently piqued interest in research that lies at the intersection of visual and textual modalities. An example of this task is demonstrated in Figure 1. It requires cross-modal reasoning to ground free- form text inside the video and calls for models ca- pable of segmenting a video into action segments (Singh et al., 2016; Yeung et al., 2016; Xu et al., 2017) as well as measuring multi-modal semantic similarity (Karpathy and Fei-Fei, 2015). This task is inherently discriminative, i.e., there is only a single most relevant clip pertaining to ∗Equal contribution, randomly ordered. 1.03 5.61 1.0 4.0 Ground Truth Predicted Figure 1: Clip Extraction task for the given query ‘the biker jumps to another ramp near the camera’ a given query in the corresponding video. How- ever, most prior works (Hendricks et al., 2017, 2018; Liu et al., 2018; Chen et al., 2018; Zhang et al., 2018) explore this as a ranking task over a fixed number of moments by uniformly sampling clips within a video. Moreover, these approaches are restrictive in scope since they use predefined clips as candidates for a video and cannot be eas- ily extended to videos with considerable variance in length. Gao et al. (2017); Xu et al. (2019) apply two- stage methods which rank candidate clips using a learned similarity metric. Gao et al. (2017); Ge et al. (2019) propose a sliding window approach with alignment and offset regression learning ob- jective, but it is limited by the coarseness of the windows and is thus inefficient and inflexible. Xu et al. (2019) address this through a query-guided segment proposal network (QSPN). However, the similarity metric used by these …

Content truncated to 500 words