On Pursuit of Designing Multi-modal Transformer for Video Grounding

On Pursuit of Designing Multi-modal Transformer for Video Grounding Meng Cao1, Long Chen2∗, Mike Zheng Shou3, Can Zhang1, and Yuexian Zou1,4† 1School of Electronic and Computer Engineering, Peking University 2Columbia University 3National University of Singapore 4Peng Cheng Laboratory mengcao@pku.edu.cn, zjuchenlong@gmail.com mike.zheng.shou@gmail.com, {zhangcan, zouyx}@pku.edu.cn Abstract Video grounding aims to localize the tempo- ral segment corresponding to a sentence query from an untrimmed video. Almost all existing video grounding methods fall into two frame- works: 1) Top-down model: It predefines a set of segment candidates and then conducts seg- ment classification and regression. 2) Bottom- up model: It directly predicts frame-wise prob- abilities of the referential segment boundaries. However, all these methods are not end-to-end, i.e., they always rely on some time-consuming post-processing steps to refine predictions. To this end, we reformulate video grounding as a set prediction task and propose a novel end-to- end multi-modal Transformer model, dubbed as GTR. Specifically, GTR has two encoders for video and language encoding, and a cross- modal decoder for grounding prediction. To fa- cilitate the end-to-end training, we use a Cubic Embedding layer to transform the raw videos into a set of visual tokens. To better fuse these two modalities in the decoder, we design a new Multi-head Cross-Modal Attention. The whole GTR is optimized via a Many-to-One match- ing loss. Furthermore, we conduct comprehen- sive studies to investigate different model de- sign choices. Extensive results on three bench- marks have validated the superiority of GTR. All three typical GTR variants achieve record- breaking performance on all datasets and met- rics, with several times faster inference speed. Our project is available at GTR. 1 Introduction Video grounding is a fundamental while challeng- ing task for video understanding and has recently attracted unprecedented research attention (Chen et al., 2018, 2019b,a; Zhang et al., 2019a; Liu et al., 2018a; Yuan et al., 2021). Formally, it aims to iden- tify the two temporal boundaries of the moment of interest based on an input untrimmed video and a ∗Work started when Long Chen at Tencent AI Lab. †Corresponding author. 5 × faster 4.9% better Figure 1: Performance comparisons on TACoS in terms of R@1, IoU@0.5 and Query Per Second (the number of queries that are retrieved each second during inference). Marker sizes are proportional to the model size. Our GTR-H is 4.9% better than 2D-TAN (Zhang et al., 2020b) with 5 times faster speed. natural language query. Compared to the conven- tional video action localization task (Shou et al., 2016; Zhao et al., 2017; Zhang et al., 2021), video grounding is more general and taxonomy-free, i.e., it is not limited by the predefined action categories. The overwhelming majority of the state-of-the- art video grounding methods fall into two frame- works: 1) Top-down models (Anne Hendricks et al., 2017; Gao et al., 2017; Ge et al., 2019; Chen et al., 2018; Zhang et al., 2019b; Liu et al., 2018b; Yuan et al., 2019a): They always use a propose-and-rank pipeline, where they first generate a set of moment proposals and …

Content truncated to 500 words