Learning to Focus on the Foreground for Temporal Sentence Grounding

Proceedings of the 29th International Conference on Computational Linguistics, pages 5532–5541 October 12–17, 2022. 5532 Learning to Focus on the Foreground for Temporal Sentence Grounding Daizong Liu and Wei Hu* Wangxuan Institute of Computer Technology, Peking University dzliu@stu.pku.edu.cn, forhuwei@pku.edu.cn Abstract Temporal sentence grounding (TSG) is crucial and fundamental for video understanding. Pre- vious works typically model the target activ- ity referred to the sentence query in a video by extracting the appearance information from each whole frame. However, these methods fail to distinguish visually similar background noise and capture subtle details of small ob- jects. Although a few recent works additionally adopt a detection model to filter out the back- ground contents and capture local appearances of foreground objects, they rely on the quality of the detection model and suffer from the time- consuming detection process. To this end, we propose a novel detection-free framework for TSG—Grounding with Learnable Foreground (GLF), which efficiently learns to locate the foreground regions related to the query in con- secutive frames for better modelling the tar- get activity. Specifically, we first split each video frame into multiple patch candidates of equal size, and reformulate the foreground de- tection problem as a patch localization task. Then, we develop a self-supervised coarse- to-fine paradigm to learn to locate the most query-relevant patch in each frame and aggre- gate them among the video for final ground- ing. Further, we employ a multi-scale patch reasoning strategy to capture more fine-grained foreground information. Extensive experiments on three challenging datasets (Charades-STA, TACoS, ActivityNet) show that the proposed GLF outperforms state-of-the-art methods. 1 Introduction Temporal sentence grounding (TSG) (Gao et al., 2017; Anne Hendricks et al., 2017) is an impor- tant yet challenging topic of video understanding in computer vision. Given an untrimmed video, *Corresponding author. Sentence Query: The person opens a bottle of wine and drinks it. Ground Truth | | 12.8s 26.3s (a) An illustration of the Temporal Sentence Grounding (TSG). ... ... Ours: Detection -based: Without Detection: ... ... ... ... ... ... ... ... ... ... Reasoning Reasoning Reasoning Reasoning Reasoning Reasoning (b) Motivation of our proposed method. Figure 1: (a) An illustrative example of the TSG task. (b) The Illustration of our motivation: we learn to se- lectively focus on the foreground patch in each frame in a detection-free manner, which alleviates the prob- lems of redundant backgrounds in most previous works (without detection) and the low detection quality in time- consuming detection-based methods. The green box marks the focused region. it aims to retrieve a temporal segment that seman- tically corresponds to a given sentence query, as shown in Figure 1 (a). Compared to other video- and-language tasks like video captioning (Song et al., 2015; Chu et al., 2015) and video action localization (Shou et al., 2016; Zhao et al., 2017; Xiong et al., 2022), TSG is substantially more chal- lenging as it need not only capture the complicated visual and textual information, but also learn the complex multi-modal interactions among them for modelling the …

Content truncated to 500 words