Memory-efficient Temporal Moment Localization in Long Videos

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1909–1924 May 2-6, 2023 ©2023 Association for Computational Linguistics Memory-efficient Temporal Moment Localization in Long Videos Cristian Rodriguez-Opazo1*, Edison Marrese-Taylor2,3* Basura Fernando4, Hiroya Takamura2, Qi Wu1 1 Australian Institute for Machine Learning 2 National Institute of Advanced Industrial Science and Technology 3 The University of Tokyo, 4 A⋆ STAR, Singapore {edison.marrese, takamura.hiroya}@aist.go.jp {cristian.rodriguezopazo, qi.wu01}@adelaide.edu.au Fernando_Basura@ihpc.a-star.edu.sg Abstract Temporal Moment Localization is a challeng- ing multimodal task which aims to identify the start and end timestamps of a moment of in- terest in an input untrimmed video, given a query in natural language. Solving this task correctly requires understanding the temporal relationships in the entire input video, but pro- cessing such long inputs and reasoning about them is memory and computationally expen- sive. In light of this issue, we propose Stochas- tic Bucket-wise Feature Sampling (SBFS), a stochastic sampling module that allows meth- ods to process long videos at a constant mem- ory footprint. We further combine SBFS with a new consistency loss to propose LOCFORMER, a Transformer-based model that can process videos as long as 18 minutes. We test our pro- posals on relevant benchmark datasets, show- ing that not only can LOCFORMER achieve excellent results, but also that our sampling is more effective than competing counterparts. Concretely, SBFS consistently improves the performance of prior work, by up to 3.13% in the mean temporal IoU, leading to a new state- of-the-art performance on Charades-STA and YouCookII, while also obtaining up to 12.8x speed-up at testing time and reducing memory requirements by up to 5x. 1 Introduction Processing long untrimmed videos for understand- ing and reasoning is a computationally expensive task that not only demands a smart approach that can capture global and local interaction, but also requires allocating thousands of frames in mem- ory (Wu et al., 2022). Previous work so far has mostly focused on parsing the visual content in in- dependent snapshots of a video, which limits the approaches in terms of modeling long-term depen- dency between events in the input. *These authors contributed equally to this work 0 30 60 90 120 180 240 300 360 420 0 8 16 24 32 40 48 56 64 Out-of-Memory Video Duration [s] GPU Memory [GB] LocFormer Transformer-base Transformer-base + sampling DORi Figure 1: Empirical upper-bound memory consumption of LOCFORMER, Transformer-base, with and without enabling our proposed sampling technique (SBFS), and DORi (Rodriguez-Opazo et al., 2021). Memory usage (y-axis) is computed on an NVIDIA RTX-8000 GPU, using a batch size of 32 and assuming all sequences have maximum length for the given video duration. In this paper, we particularly look at the task of temporal moment localization, which aims to iden- tify the start and end timestamps of a moment of in- terest in an input untrimmed video given the query in natural language (Richard et al., 2018; Lin et al., 2017; Escorcia et al., 2016; Chao et al., 2018; Gao et al., 2017b; Xu …

Content truncated to 500 words