Reasoning Step-by-Step: Temporal Sentence Localization in Videos via Deep Rectification-Modulation Network

Proceedings of the 28th International Conference on Computational Linguistics, pages 1841–1851 Barcelona, Spain (Online), December 8-13, 2020 1841 Reasoning Step-by-Step: Temporal Sentence Localization in Videos via Deep Rectification-Modulation Network Daizong Liu1, Xiaoye Qu2, Jianfeng Dong3, Pan Zhou1* 1Huazhong University of Science and Technology 2Huawei Cloud 3Zhejiang Gongshang University dzliu@hust.edu.cn, quxiaoye@huawei.com, dongjf24@gmail.com, panzhou@hust.edu.cn Abstract Temporal sentence localization in videos aims to ground the best matched segment in an untrimmed video according to a given sentence query. Previous works in this field mainly rely on single-step attentional frameworks to align the temporal boundaries by a soft selection. Although they focus on the visual content relevant to the query, these attention strategies are insufficient to model complex video contents and restrict the higher-level reasoning demand for temporal relation. In this paper, we propose a novel deep rectification-modulation network (RMN), trans- forming this task into a multi-step reasoning process by repeating rectification and modulation. In each rectification-modulation layer, unlike existing methods directly conducting the cross-modal interaction, we first devise a rectification module to correct implicit attention misalignment which focuses on wrong position during the interaction process. Then, a modulation module is devel- oped to model the frame-to-frame relation with the help of specific sentence information for bet- ter correlating and composing the video contents over time. With multiple such layers cascaded in depth, our RMN progressively refines video and query interactions, thus enabling a further precise localization. Experimental evaluations on three public datasets show that the proposed method achieves state-of-the-art performance. 1 Introduction Localizing activities in videos (Regneri et al., 2013; Yuan et al., 2016; Gavrilyuk et al., 2018; Feng et al., 2018; Feng et al., 2019) is an important topic in information retrieval systems. As most videos contain activities of interest with complicated background contents, these videos cannot be directly indicated by a pre-defined list of action classes. Recently, a new task called temporal sentence localization in videos (Gao et al., 2017; Anne Hendricks et al., 2017) is proposed to tackle this problem, attracting great interests from both vision and language communities (Liu et al., 2020; Qu et al., 2020). Given an untrimmed video, this task aims to infer the start and end timestamps of a target video segment which contains the interested activity according to a given sentence query. Traditional methods (Gao et al., 2017; Liu et al., 2018; Ge et al., 2019; Chen and Jiang, 2019; Anne Hendricks et al., 2017) are based on sliding windows, which first sample candidate video segments and then compare the sentence with each video segment separately to calculate the matching relation- ships. These methods cannot achieve precise alignment between video and sentence, thus leading to inaccurate temporal boundaries. Recently, some works (Chen et al., 2018; Chen et al., 2019; Zhang et al., 2019b; Zhang et al., 2019a) try to avoid this problem by designing end-to-end models. They first integrate the features of the whole video with sentence information and then utilize LSTM or CNN layer to compose such integrated video features for further segment localization. …

Content truncated to 500 words