Natural Language Video Localization with Learnable Moment Proposals

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4008–4017 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 4008 Natural Language Video Localization with Learnable Moment Proposals Shaoning Xiao†, Long Chen‡∗, Jian Shao†, Yueting Zhuang†, and Jun Xiao† †DCD Lab, College of Computer Science, Zhejiang University ‡DVMM Lab, Columbia University shaoningx@zju.edu.cn, zjuchenlong@gmail.com jshao@zju.edu.cn, yzhuang@zju.edu.cn, junx@zju.edu.cn Abstract Given an untrimmed video and a natural lan- guage query, Natural Language Video Local- ization (NLVL) aims to identify the video mo- ment described by the query. To address this task, existing methods can be roughly grouped into two groups: 1) propose-and-rank mod- els first define a set of hand-designed moment candidates and then find out the best-matching one. 2) proposal-free models directly predict two temporal boundaries of the referential mo- ment from frames. Currently, almost all the propose-and-rank methods have inferior per- formance than proposal-free counterparts. In this paper, we argue that propose-and-rank ap- proach is underestimated due to the predefined manners: 1) Hand-designed rules are hard to guarantee the complete coverage of targeted segments. 2) Densely sampled candidate mo- ments cause redundant computation and de- grade the performance of ranking process. To this end, we propose a novel model termed LP- Net (Learnable Proposal Network for NLVL) with a fixed set of learnable moment propos- als. The position and length of these proposals are dynamically adjusted during training pro- cess. Moreover, a boundary-aware loss has been proposed to leverage frame-level infor- mation and further improve the performance. Extensive ablations on two challenging NLVL benchmarks have demonstrated the effective- ness of LPNet over existing state-of-the-art methods1. 1 Introduction Natural Language Video Localization (NLVL), aka, video grounding or video moment localization, has got unprecedented attention in both CV and NLP communities (Gao et al., 2017; Hendricks et al., 2017). As shown in Figure 1, NLVL aims to local- ∗Long Chen is the corresponding author. The work started when Long Chen was at Zhejiang University. 1Source codes is available: https://github.com/ xiaoneil/LPNet/ Language Query: The lady spins the stick using her neck. 66.79s 85.34s Ground Truth: Figure 1: An illustrative example of NLVL. Given a video and a query, NLVL is to localize the video seg- ment corresponding to the query with the start times- tamp (66.79s) and the end timestamp (85.34s). ize the video segment relevant to the query by locat- ing the start and end timestamps in an untrimmed video. It is challenging since it needs to not only un- derstand the video and the sentence content but also find out the precise temporal boundaries. Moreover, NLVL is helpful to numerous downstream video understanding tasks, e.g., content retrieval (Shao et al., 2018), relation detection (Gao et al., 2021), and VQA (Lei et al., 2018; Ye et al., 2017). Currently, state-of-the-art NLVL methods can be roughly grouped into two categories according to how the video segments are detected, namely propose-and-rank and proposal-free methods: The idea of the propose-and-rank approach (Gao et al., 2017; Hendricks et al., 2018; Liu et al., 2018b; …

Content truncated to 500 words