Adaptive Proposal Generation Network for Temporal Sentence Localization in Videos

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9292–9301 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 9292 Adaptive Proposal Generation Network for Temporal Sentence Localization in Videos Daizong Liu1,2*, Xiaoye Qu3*, Jianfeng Dong4, Pan Zhou1† 1The Hubei Engineering Research Center on Big Data Security, School of Cyber Science and Engineering, Huazhong University of Science and Technology 2School of Electronic Information and Communication, Huazhong University of Science and Technology 3Huawei Cloud 4Zhejiang Gongshang University {dzliu,panzhou}@hust.edu.cn, quxiaoye@huawei.com, dongjf24@gmail.com Abstract We address the problem of temporal sen- tence localization in videos (TSLV). Tradi- tional methods follow a top-down framework which localizes the target segment with pre- defined segment proposals. Although they have achieved decent performance, the pro- posals are handcrafted and redundant. Re- cently, bottom-up framework attracts increas- ing attention due to its superior efficiency. It di- rectly predicts the probabilities for each frame as a boundary. However, the performance of bottom-up model is inferior to the top-down counterpart as it fails to exploit the segment- level interaction. In this paper, we pro- pose an Adaptive Proposal Generation Net- work (APGN) to maintain the segment-level interaction while speeding up the efficiency. Specifically, we first perform a foreground- background classification upon the video and regress on the foreground frames to adaptively generate proposals. In this way, the hand- crafted proposal design is discarded and the redundant proposals are decreased. Then, a proposal consolidation module is further devel- oped to enhance the semantic of the generated proposals. Finally, we locate the target mo- ments with these generated proposals follow- ing the top-down framework. Extensive exper- iments on three challenging benchmarks show that our proposed APGN significantly outper- forms previous state-of-the-art methods. 1 Introduction Temporal sentence localization in videos is an im- portant yet challenging task in natural language processing, which has drawn increasing attention over the last few years due to its vast potential applications in information retrieval (Dong et al., 2019; Yang et al., 2020) and human-computer inter- action (Singha et al., 2018). It aims to ground the most relevant video segment according to a given ⇤Equal contributions. †Corresponding author. Query: He continues playing the instrument while pausing to speak to the camera. Start:138.65s | | End:215.94s (b) Top-Down framework (c) Bottom-Up framework GT (a) An example of temporal sentence localization in video pre-defined proposals Video-Query interacted representation frame-wise probabilities Video-Query interacted representation start end Figure 1: (a) An example of temporal sentence localiza- tion in videos. (b) The Top-Down framework predicts the confidence scores of a large number of pre-defined proposals for ranking. (c) The Bottom-Up framework regresses the probabilities of all frames as start or end boundaries. sentence query. As shown in Figure 1 (a), most parts of video contents are irrelevant to the query (background) while only a short segment matches it (foreground). Therefore, video and query informa- tion need to be deeply incorporated to distinguish the fine-grained details of different video segments. Most previous works (Gao et al., 2017; Chen et al., 2018; Zhang …

Content truncated to 500 words