End-to-End Modeling via Information Tree for One-Shot Natural Language Spatial Video Grounding

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 8707 - 8717 May 22-27, 2022 c⃝2022 Association for Computational Linguistics End-to-End Modeling via Information Tree for One-Shot Natural Language Spatial Video Grounding Mengze Li1, Tianbao Wang1, Haoyu Zhang1, Shengyu Zhang1, Zhou Zhao2,3∗, Jiaxu Miao1∗, Wenqiao Zhang1∗, Wenming Tan4, Jin Wang4, Peng Wang5, Shiliang Pu4, Fei Wu2,3∗ 1Zhejiang University 2Shanghai Institute for Advanced Study of Zhejiang University 3Shanghai AI Laboratory 4Hikvision Research Institute 5Northwestern Polytechnical University Abstract Natural language spatial video grounding aims to detect the relevant objects in video frames with descriptive sentences as the query. In spite of the great advances, most existing meth- ods rely on dense video frame annotations, which require a tremendous amount of hu- man effort. To achieve effective grounding under a limited annotation budget, we inves- tigate one-shot video grounding, and learn to ground natural language in all video frames with solely one frame labeled, in an end-to- end manner. One major challenge of end-to- end one-shot video grounding is the existence of videos frames that are either irrelevant to the language query or the labeled frames. An- other challenge relates to the limited super- vision, which might result in ineffective rep- resentation learning. To address these chal- lenges, we designed an end-to-end model via Information Tree for One-Shot video ground- ing (IT-OS). Its key module, the information tree, can eliminate the interference of irrele- vant frames based on branch search and branch cropping techniques. In addition, several self- supervised tasks are proposed based on the in- formation tree to improve the representation learning under insufficient labeling. Experi- ments on the benchmark dataset demonstrate the effectiveness of our model. 1 Introduction Natural language spatial video grounding is a vi- tal task for video-text understanding (Luo and Shakhnarovich, 2017; Zhou et al., 2019; Hu et al., 2019; Zhang et al., 2020b; Li et al., 2021), which aims to detect the objects described by the natural First author. mengzeli@zju.edu.cn ∗Corresponding author. zhaozhou@zju.edu.cn jiaxu.miao@yahoo wenqiaozhang@zju.edu.cn wufei@zju.edu.cn Figure 1: An example of spatially grounding natural language in video frames. language query from each video frame, as shown in Figure 1. There is a substantial and rapidly- growing research literature studying this problem with dense annotations (Li et al., 2017; Yamaguchi et al., 2017; Sadhu et al., 2020), where each frame that contains objects relevant to the language query will be manually labeled with bounding boxes. Ob- viously, such annotations require tremendous hu- man effort and can hardly be satisfied in real-world scenarios. Recently, some works have investigated weakly-supervised video grounding with solely the video-text correspondence rather than object-text annotations (Huang et al., 2018; Chen et al., 2019a; Shi et al., 2019; Chen et al., 2019b; Zhou et al., 2018). However, the performance is less satisfied with such weak supervision. In practice, we are more likely to have a limited annotation budget rather than full annotation or no annotation. In addition, as humans, after experiencing the lan- guage query and one …

Content truncated to 500 words