Relation-aware Video Reading Comprehension for Temporal Language Grounding

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3978–3988 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 3978 Relation-aware Video Reading Comprehension for Temporal Language Grounding Jialin Gao1,2∗ Xin Sun1,2∗ MengMeng Xu3 Xi Zhou1,2 Bernard Ghanem3 1Cooperative Medianet Innovation Center, Shanghai Jiao Tong University 2 CloudWalk Technology Co., Ltd, China 3 King Abdullah University of Science and Technology {jialin_gao, huntersx}@sjtu.edu.cn, zhouxi@cloudwalk.cn {mengmeng.xu,bernard.ghanem}@kaust.edu.sa Abstract Temporal language grounding in videos aims to localize the temporal span relevant to the given query sentence. Previous methods treat it either as a boundary regression task or a span extraction task. This paper will formulate tem- poral language grounding into video reading comprehension and propose a Relation-aware Network (RaNet) to address it. This frame- work aims to select a video moment choice from the predefined answer set with the aid of coarse-and-fine choice-query interaction and choice-choice relation construction. A choice- query interactor is proposed to match the vi- sual and textual information simultaneously in sentence-moment and token-moment lev- els, leading to a coarse-and-fine cross-modal interaction. Moreover, a novel multi-choice relation constructor is introduced by leverag- ing graph convolution to capture the dependen- cies among video moment choices for the best choice selection. Extensive experiments on ActivityNet-Captions, TACoS, and Charades- STA demonstrate the effectiveness of our so- lution. Codes will be available at https: //github.com/Huntersxsx/RaNet. 1 Introduction Recently, temporal language grounding in videos has become a heated topic in the computer vision, and natural language processing community (Gao et al., 2017; Krishna et al., 2017). This task requires a machine to localize a temporal moment semanti- cally relevant to a given language query, as shown in Fig.1. It has also drawn great attention from in- dustry due to its various applications such as video question answering (Huang et al., 2020; Lei et al., 2018), video content retrieval (Dong et al., 2019; Shao et al., 2018), and human-computer interaction (Zhu et al., 2020), etc. A straightforward paradigm for this task is the proposing-and-ranking pipelines (Xu et al., 2019; ∗ Jialin Gao and Xin Sun are co-first authors with equal contributions, supervised by Prof. Xi Zhou in SJTU. Figure 1: An illustration of temporal language grounding in videos based on the relation-aware net- work. Given a video and a query sentence, our ap- proach aims to semantically align the query represen- tation with a predefined answer set of video moment candidates (a1,a2,a3 and a4) and then mine the rela- tionships between them to select the best-matched one. Zhang et al., 2020b, 2019a). They first generate a number of video moment candidates and then rank them according to moment-query similarities. This requires a solution to achieve two key targets simul- taneously, which are (1) semantic visual-language interaction and (2) reliable candidate ranking. The former ensures a satisfying cross-modal match- ing between video moments and the query, while the latter guarantees the distinction among candi- dates. For the first target, some previous works (Yuan et al., 2019; Zhang et al., 2020b; Chen and Jiang, 2019) resort …

Content truncated to 500 words