CONE: An Efficient COarse-to-fiNE Alignment Framework for Long Video Temporal Grounding

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 8013–8028 July 9-14, 2023 ©2023 Association for Computational Linguistics CONE: An Efficient COarse-to-fiNE Alignment Framework for Long Video Temporal Grounding Zhijian Hou1∗, Wanjun Zhong2∗, Lei Ji3, Difei Gao4, Kun Yan5, Wing-Kwong Chan1, Chong-Wah Ngo6, Mike Zheng Shou4 and Nan Duan3 1 City University of Hong Kong 2 Sun Yat-sen University 3 Microsoft Research Asia 4 National University of Singapore 5 Beihang University 6 Singapore Management University {zjhou3-c@my.,wkchan@}cityu.edu.hk; zhongwj25@mail2.sysu.edu.cn; {leiji,nanduan}@microsoft.com difei.gao@vipl.ict.ac.cn; kunyan@buaa.edu.cn; cwngo@smu.edu.sg; mike.zheng.shou@gmail.com Abstract This paper tackles an emerging and challeng- ing problem of long video temporal ground- ing (VTG) that localizes video moments related to a natural language (NL) query. Compared with short videos, long videos are also highly- demanded but less explored, which brings new challenges in higher inference computation cost and weaker multi-modal alignment. To address these challenges, we propose CONE, an ef- ficient COarse-to-fiNE alignment framework. CONE is a plug-and-play framework on top of existing VTG models to handle long videos through a sliding window mechanism. Specif- ically, CONE (1) introduces a query-guided window selection strategy to speed up infer- ence, and (2) proposes a coarse-to-fine mech- anism via a novel incorporation of contrastive learning to enhance multi-modal alignment for long videos. Extensive experiments on two large-scale long VTG benchmarks consistently show both substantial performance gains (e.g., 3.13% +119% −−−→6.87% on MAD) and state-of-the- art results. Analyses also reveal higher effi- ciency as the query-guided window selection mechanism accelerates inference time by 2x on Ego4D-NLQ and 15x on MAD while keep- ing SOTA results. Codes have been released at https://github.com/houzhijian/CONE. 1 Introduction Video temporal grounding (Anne Hendricks et al., 2017; Gao et al., 2017) aims to locate specific mo- ments in an untrimmed video relevant to a textual user query. This is a crucial task in multi-modal video understanding and can be applied to many practical applications such as video retrieval (Xu et al., 2016), video editing, and video question an- swering (Lei et al., 2018, 2020a). Temporal grounding for long videos is espe- cially highly-demanded and emerging due to the ∗ Indicates equal contribution. This work was done during the first and second authors’ internships in MSR Asia. NL query In the room … TimeLine: 1200 s 60 s Coarse Level Not holding No woman Not holding No woman GT Answer Video 0 s Fine Level Outdoor 21 s 26 s 21 s 26 s : What object was the woman in the room holding? answer Figure 1: An example of long video temporal grounding, which requires coarse-to-fine multi-modal alignment. flourishing growth of online videos in quantity and length. However, limited by previously available datasets (Gao et al., 2017; Krishna et al., 2017; Lei et al., 2021), existing works (Zhang et al., 2020a,b) mainly deal with relatively short videos ranging from 0.5 to 2.5 minutes on average. Recently, Ego4D (Grauman et al., 2022) and MAD (Sol- dan et al., 2022) datasets have been created and attempted to deal …

Content truncated to 500 words