Modal-specific Pseudo Query Generation for Video Corpus Moment Retrieval

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7769 - 7781 December 7-11, 2022 ©2022 Association for Computational Linguistics Modal-specific Pseudo Query Generation for Video Corpus Moment Retrieval Minjoon Jung1, Seongho Choi1, Joochan Kim1, Jin-Hwa Kim2,3∗, and Byoung-Tak Zhang1,3∗ 1Seoul National University 2NAVER AI Lab 3AI Institute of Seoul National University {mjjung, shchoi, jckim}@bi.snu.ac.kr, j1nhwa.kim@navercorp.com, btzhang@bi.snu.ac.kr Abstract Video corpus moment retrieval (VCMR) is the task to retrieve the most relevant video moment from a large video corpus using a natural lan- guage query. For narrative videos, e.g., dramas or movies, the holistic understanding of tem- poral dynamics and multimodal reasoning is crucial. Previous works have shown promising results; however, they relied on the expensive query annotations for VCMR, i.e., the corre- sponding moment intervals. To overcome this problem, we propose a self-supervised learning framework: Modal-specific Pseudo Query Gen- eration Network (MPGN). First, MPGN selects candidate temporal moments via subtitle-based moment sampling. Then, it generates pseudo queries exploiting both visual and textual infor- mation from the selected temporal moments. Through the multimodal information in the pseudo queries, we show that MPGN success- fully learns to localize the video corpus mo- ment without any explicit annotation. We val- idate the effectiveness of MPGN on the TVR dataset, showing competitive results compared with both supervised models and unsupervised setting models. 1 Introduction The increased interest in video understanding has gathered attention for solving related tasks such as video captioning (Krishna et al., 2017), video question answering (Tapaswi et al., 2016; Lei et al., 2018; Kim et al., 2018), and video retrieval (Xu et al., 2016) over the past few years. Video corpus moment retrieval (VCMR) (Escorcia et al., 2019) is one of the challenging video understanding tasks, in which a model should 1) search for a related video and 2) localize the corresponding moment given a query sentence in a large video corpus. Prior works have shown promising performances in VCMR using supervised (Lei et al., 2020; Zhang et al., 2020, 2021), weakly-supervised (Yoon et al., ∗ Corresponding authors. Figure 1: Comparison with different supervision set- tings in VCMR. (a) Fully-supervised VCMR (paired video query with timestamp), (b) Weakly-supervised VCMR (paired video query without timestamp), (c) VCMR without annotation (unlabeled video). 2021) settings, and pre-training (Li et al., 2020; Zhou et al., 2021) methods. Despite such accom- plishments, selecting a temporal moment in a video (start time, end time) and generating the corre- sponding query sentence to train such models re- quire overwhelming amounts of human labor. To annotate these videos, humans first need to under- stand the diverse information in the video, and then select the candidate temporal moment and generate corresponding queries. The challenges of this VCMR are two folds: 1) Considering a large number of human annotations are required, an efficient approach is required to reduce the annotation cost. 2) These multimodal videos (e.g., drama or movies) generally contain rich interactions between characters, which widely 7769 exist but has rarely been studied in the VCMR …

Content truncated to 500 words