Span-based Localizing Network for Natural Language Video Localization

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6543–6554 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 6543 Span-based Localizing Network for Natural Language Video Localization Hao Zhang1,2, Aixin Sun1, Wei Jing2,3, Joey Tianyi Zhou2,∗ 1School of Computer Science and Engineering, Nanyang Technological University, Singapore 2Institute of High Performance Computing, A*STAR, Singapore 3Institute for Infocomm Research, A*STAR, Singapore hao007@e.ntu.edu.sg, axsun@ntu.edu.sg 21wjing@gmail.com, joey zhou@ihpc.a-star.edu.sg Abstract Given an untrimmed video and a text query, natural language video localization (NLVL) is to locate a matching span from the video that semantically corresponds to the query. Exist- ing solutions formulate NLVL either as a rank- ing task and apply multimodal matching ar- chitecture, or as a regression task to directly regress the target video span. In this work, we address NLVL task with a span-based QA approach by treating the input video as text passage. We propose a video span localiz- ing network (VSLNet), on top of the standard span-based QA framework, to address NLVL. The proposed VSLNet tackles the differences between NLVL and span-based QA through a simple and yet effective query-guided high- lighting (QGH) strategy. The QGH guides VSLNet to search for matching video span within a highlighted region. Through exten- sive experiments on three benchmark datasets, we show that the proposed VSLNet outper- forms the state-of-the-art methods; and adopt- ing span-based QA framework is a promising direction to solve NLVL.1 1 Introduction Given an untrimmed video, natural language video localization (NLVL) is to retrieve or localize a tem- poral moment that semantically corresponds to a given language query. An example is shown in Figure 1. As an important vision-language under- standing task, NLVL involves both computer vision and natural language processing techniques (Kr- ishna et al., 2017; Hendricks et al., 2017; Gao et al., 2018; Le et al., 2019; Yu et al., 2019). Clearly, cross-modal reasoning is essential for NLVL to correctly locate the target moment from a video. Prior works primarily treat NLVL as a rank- ing task, which is solved by applying multimodal ∗Corresponding author. 1https://github.com/IsaacChanghau/VSLNet Language Query: Men are celebrating and an old man gives a trophy to a young boy. Timeline (second) 127.52 139.20 0.00 194.69 The Ground Truth Moment Figure 1: An illustration of localizing a temporal moment in an untrimmed video by a given language query. matching architecture to find the best matching video segment for a given language query (Gao et al., 2017; Hendricks et al., 2018; Liu et al., 2018a; Ge et al., 2019; Xu et al., 2019; Chen and Jiang, 2019; Zhang et al., 2019). Recently, some works explore to model cross-interactions between video and query, and to regress the temporal loca- tions of target moment directly (Yuan et al., 2019b; Lu et al., 2019a). There are also studies to formu- late NLVL as a sequence decision making problem and to solve it by reinforcement learning (Wang et al., 2019; He et al., 2019). We address the NLVL task from a different per- spective. The …

Content truncated to 500 words