Parallel Attention Network with Sequence Matching for Video Grounding

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 776–790 August 1–6, 2021. ©2021 Association for Computational Linguistics 776 Parallel Attention Network with Sequence Matching for Video Grounding Hao Zhang1,2, Aixin Sun1, Wei Jing2,3 Liangli Zhen2, Joey Tianyi Zhou2, Rick Siow Mong Goh2 1School of Computer Science and Engineering, Nanyang Technological University, Singapore 2Institute of High Performance Computing, A*STAR, Singapore 3Institute for Infocomm Research, A*STAR, Singapore {hao007@e.,axsun@}ntu.edu.sg, 21wjing@gmail.com, {zhenll,zhouty,gohsm}@ihpc.a-star.edu.sg Abstract Given a video, video grounding aims to re- trieve a temporal moment that semantically corresponds to a language query. In this work, we propose a Parallel Attention Network with Sequence matching (SeqPAN) to address the challenges in this task: multi-modal represen- tation learning, and target moment boundary prediction. We design a self-guided parallel attention module to effectively capture self- modal contexts and cross-modal attentive in- formation between video and text. Inspired by sequence labeling tasks in natural language processing, we split the ground truth moment into begin, inside, and end regions. We then propose a sequence matching strategy to guide start/end boundary predictions using region la- bels. Experimental results on three datasets show that SeqPAN is superior to state-of-the- art methods. Furthermore, the effectiveness of the self-guided parallel attention module and the sequence matching module is verified.1 1 Introduction Video grounding is a fundamental and challenging problem in vision-language understanding research area (Hu et al., 2019; Yu et al., 2019; Zhu and Yang, 2020). It aims to retrieve a temporal video moment that semantically corresponds to a given language query, as shown in Figure 1. This task re- quires techniques from both computer vision (Tran et al., 2015; Shou et al., 2016; Feichtenhofer et al., 2019), natural language processing (Yu et al., 2018; Yang et al., 2019), and more importantly, the cross- modal interactions between the two. Many existing solutions (Chen et al., 2018; Liu et al., 2018a; Xu et al., 2019) tackle video grounding problem with proposal-based approach. This approach generates proposals with pre-set sliding windows or anchors, computes the similarity between the query and each 1https://github.com/IsaacChanghau/SeqPAN Timeline 0.00s 136.26s 57.23s Query: They continue forward very slowly taking their time and enjoying the experience. Target Moment Video Feature Extraction Integration Localization Self-Attention Self-Attention 90.61s Cross-Attention Cross-Attention Cross Gating Figure 1: The overview of our procedures for video ground- ing, with an example of retrieving the temporal moment from an untrimmed video by a given language query. proposal. The proposal with highest score is se- lected as the answer. These methods are sensitive to the quality of proposals and are inefficient because all proposal-query pairs are compared. Recently, several one-stage proposal-free solutions (Chen et al., 2019; Lu et al., 2019a; Mun et al., 2020) are proposed to directly predict start/end boundaries of target moments, through modeling video-text inter- actions. Our solution, SeqPAN, is a proposal-free method; hence our key focuses are video-text inter- action modeling and moment boundary prediction. Video-text interaction modeling. In order to model video-text interaction, various attention- based methods have been proposed (Gao et al., 2017; …

Content truncated to 500 words