Generating Structured Pseudo Labels for Noise-resistant Zero-shot Video Sentence Localization

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 14197–14209 July 9-14, 2023 ©2023 Association for Computational Linguistics Generating Structured Pseudo Labels for Noise-resistant Zero-shot Video Sentence Localization Minghang Zheng1, Shaogang Gong2, Hailin Jin3, Yuxin Peng1, 4, and Yang Liu1,5∗ 1Wangxuan Institute of Computer Technology, Peking University 2Queen Mary University of London, 3Adobe Research 4 National Key Laboratory for Multimedia Information Processing, Peking University 5 National Key Laboratory of General Artificial Intelligence, BIGAI {minghang, pengyuxin, yangliu}@pku.edu.cn s.gong@qmul.ac.uk, hljin@adobe.com Abstract Video sentence localization aims to locate mo- ments in an unstructured video according to a given natural language query. A main challenge is the expensive annotation costs and the anno- tation bias. In this work, we study video sen- tence localization in a zero-shot setting, which learns with only video data without any annota- tion. Existing zero-shot pipelines usually gener- ate event proposals and then generate a pseudo query for each event proposal. However, their event proposals are obtained via visual fea- ture clustering, which is query-independent and inaccurate; and the pseudo-queries are short or less interpretable. Moreover, existing ap- proaches ignores the risk of pseudo-label noise when leveraging them in training. To address the above problems, we propose a Structure- based Pseudo Label generation (SPL), which first generate free-form interpretable pseudo queries before constructing query-dependent event proposals by modeling the event tempo- ral structure. To mitigate the effect of pseudo- label noise, we propose a noise-resistant itera- tive method that repeatedly re-weight the train- ing sample based on noise estimation to train a grounding model and correct pseudo labels. Experiments on the ActivityNet Captions and Charades-STA datasets demonstrate the advan- tages of our approach. Code can be found at https://github.com/minghangz/SPL. 1 Introduction Video sentence localization, which aims to localize the most salient video segments from an untrimmed video given a free-form nature language query, has attained increasing attention due to its potential applications in video surveillance (Collins et al., 2000), robot manipulation (Kemp et al., 2007), etc. The free-form natural language queries allow the model to be flexibly adapted to the requirements of different practical applications. ∗Corresponding author Zero-shot Video: Step 1. Pseudo Query: A man cleans the snow with a brush Similarity Time Step 3. Pseudo Event: [4s, 15s] Step 1. Pseudo Event: [8s, 11s] Step 2. Pseudo Query: Person cleans the snow Similarity Time Unaligned (c) Existing pipeline (d) Ours pipeline Video: Aligned Query: A man cleans the snow on the car GT : … … Fully Supervised Weakly Supervised Video: [3s, 14.5s] Query: A man cleans the snow on the car … … Video: (a) (b) Step 2. Similarity: Figure 1: (a) Training data for fully-supervised models. (b) Training data for weakly-supervised models. (c) The zero-shot models are trained with videos only. Existing pipeline may generate unaligned pseudo event-query pairs. (d) We construct query-dependent event proposals by modeling the event temporal structure. In recent years, the performance of video sen- tence localization has been improved with the help of …

Content truncated to 500 words