Weakly-Supervised Spoken Video Grounding via Semantic Interaction Learning

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 10914–10932 July 9-14, 2023 ©2023 Association for Computational Linguistics Weakly-Supervised Spoken Video Grounding via Semantic Interaction Learning Ye Wang∗, Wang Lin∗, Shengyu Zhang∗, Tao Jin, Linjun Li, Xize Cheng, Zhou Zhao† Zhejiang University {yew,linwanglw,sy_zhang}@zju.edu.cn {jint_zju,lilinjun21,chengxize,zhaozhou}@zju.edu.cn Abstract The task of spoken video grounding aims to localize moments in videos that are relevant to descriptive spoken queries. However, ex- tracting semantic information from speech and modeling the cross-modal correlation pose two critical challenges. Previous studies solve them by representing spoken queries based on the matched video frames, which require tremen- dous effort for frame-level labeling. In this work, we investigate weakly-supervised spo- ken video grounding, i.e., learning to localize moments without expensive temporal annota- tions. To effectively represent the cross-modal semantics, we propose Semantic Interaction Learning (SIL), a novel framework consisting of the acoustic-semantic pre-training (ASP) and acoustic-visual contrastive learning (AVCL). In ASP, we pre-train an effective encoder for the grounding task with three comprehensive tasks, where the robustness task enhances stability by explicitly capturing the invariance between time- and frequency-domain features, the con- ciseness task avoids over-smooth attention by compressing long sequence into segments, and the semantic task improves spoken language understanding by modeling the precise seman- tics. In AVCL, we mine pseudo labels with discriminative sampling strategies and directly strengthen the interaction between speech and video by maximizing their mutual information. Extensive experiments demonstrate the effec- tiveness and superiority of our method.1 1 Introduction Temporal video grounding (Gao et al., 2017; Hen- dricks et al., 2017) is an important task in the cross- modal understanding field (Zhang et al., 2020d; Jin et al., 2020; Xun et al., 2021; Jin and Zhao, 2021; Yin et al., 2022), aiming to retrieve a target moment within a video based on a given query. ∗ Equal contribution. † Corresponding author 1https://github.com/yewzz/SIL. Figure 1: An Example of Spoken Video Grounding. With the progress in deep learning, there has been significant achievements in this area. While most previous studies focus on textual queries, recent work (Xia et al., 2022) introduce the spoken video grounding task by incorporating the spoken query into the video grounding, as shown in Figure 1. However, such video grounding task with spo- ken queries presents unique challenges compared to its text-based counterpart. First, the encoding of speech is inherently difficult due to its weak and volatile semantic information, making it ardu- ous to extract useful features for video grounding. Second, even with the acquisition of valuable se- mantic features, modeling speech-video interaction and extracting cross-modal content still poses an inevitable obstacle. Prior work (Xia et al., 2022) address these two problems simultaneously through the proposed video-guided contrastive predictive coding, which utilizes aligned video clips to learn semantic repre- sentations from spoken queries. However, a critical drawback is its heavy reliance on precise tempo- ral matching between video and spoken queries. The acquisition of such fine-grained annotations requires substantial manual labor, hindering the practicality and applicability …

Content truncated to 500 words