DEBUG: A Dense Bottom-Up Grounding Approach for Natural Language Video Localization

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 5144–5153, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 5144 DEBUG: A Dense Bottom-Up Grounding Approach for Natural Language Video Localization Chujie Lu1∗ Long Chen1∗ Chilie Tan2 Xiaolin Li2,3 Jun Xiao1† 1DCD Lab, College of Computer Science, Zhejiang University, China 2Tongdun Technology, China 3University of Florida, USA chujielu@outlook.com, {longc, junx}@zju.edu.cn chilie.tan@tongdun.net, andyli@ece.ufl.edu Abstract In this paper, we focus on natural language video localization: localizing (i.e., grounding) a natural language description in a long and untrimmed video sequence. All currently pub- lished models for addressing this problem can be categorized into two types: (i) top-down approach: it does classification and regres- sion for a set of pre-cut video segment candi- dates; (ii) bottom-up approach: it directly pre- dicts probabilities for each video frame as the temporal boundaries (i.e., start and end time point). However, both two approaches suffer several limitations: the former is computation- intensive for densely placed candidates, while the latter has trailed the performance of the top-down counterpart thus far. To this end, we propose a novel dense bottom-up framework: DEnse Bottom-Up Grounding (DEBUG). DE- BUG regards all frames falling in the ground truth segment as foreground, and each fore- ground frame regresses the unique distances from its location to bi-directional ground truth boundaries. Extensive experiments on three challenging benchmarks (TACoS, Charades- STA, and ActivityNet Captions) show that DE- BUG is able to match the speed of bottom-up models while surpassing the performance of the state-of-the-art top-down models. 1 Introduction Vision-and-language understanding, e.g., what the vision and text are, and how they relate with each other, is one of the core tasks in both computer vi- sion and natural language processing. To test the machine comprehension of complex video scene and natural language simultaneously, a challeng- ing task was proposed (Gao et al., 2017; Hen- dricks et al., 2017, 2018), called Natural Lan- guage Video Localization (NLVL). As shown ∗ Chujie Lu and Long Chen are co-first authors with equal contributions. † Jun Xiao is the corresponding author. Language Query: People are scrubbing the ice in front of a ball. 89.7s 94.2s Figure 1: Natural language video localization is to lo- calize a segment with the start point (89.7s) and end point (94.2s) in the video given a language description. in Figure 1, given a natural language description query and an untrimmed video sequence, NLVL needs to localize a segment in the video (i.e., iden- tify the start and end time point of the segment) which semantically corresponds to the reference sentence. Moreover, NLVL is an indispensable technique for many important applications, e.g., text-oriented video highlight detection or retrieval. Currently, the overwhelming majority of NLVL models are top-down approaches: they first cut a video into a set of segment candidates, then they do classification and regression for each candidate. Specifically, they can be further grouped into two sub-types: 1) sliding-window-based (Gao et al., …

Content truncated to 500 words