Progressively Guide to Attend: An Iterative Alignment Framework for Temporal Sentence Grounding

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9302–9311 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 9302 Progressively Guide to Attend: An Iterative Alignment Framework for Temporal Sentence Grounding Daizong Liu1,2, Xiaoye Qu3, Pan Zhou1* 1The Hubei Engineering Research Center on Big Data Security, School of Cyber Science and Engineering, Huazhong University of Science and Technology 2School of Electronic Information and Communication, Huazhong University of Science and Technology 3Huawei Cloud dzliu@hust.edu.cn, quxiaoye@huawei.com, panzhou@hust.edu.cn Abstract A key solution to temporal sentence grounding (TSG) exists in how to learn effective align- ment between vision and language features ex- tracted from an untrimmed video and a sen- tence description. Existing methods mainly leverage vanilla soft attention to perform the alignment in a single-step process. However, such single-step attention is insufficient in practice, since complicated relations between inter- and intra-modality are usually obtained through multi-step reasoning. In this paper, we propose an Iterative Alignment Network (IA-Net) for TSG task, which iteratively in- teracts inter- and intra-modal features within multiple steps for more accurate grounding. Specifically, during the iterative reasoning pro- cess, we pad multi-modal features with learn- able parameters to alleviate the nowhere-to- attend problem of non-matched frame-word pairs, and enhance the basic co-attention mech- anism in a parallel manner. To further cali- brate the misaligned attention caused by each reasoning step, we also devise a calibration module following each attention module to re- fine the alignment knowledge. With such it- erative alignment scheme, our IA-Net can ro- bustly capture the fine-grained relations be- tween vision and language domains step-by- step for progressively reasoning the temporal boundaries. Extensive experiments conducted on three challenging benchmarks demonstrate that our proposed model performs better than the state-of-the-arts. 1 Introduction Temporal localization is an important topic of vi- sual understanding in computer vision. There are several related tasks proposed for different scenar- ios involving language, such as video summariza- tion (Song et al., 2015; Chu et al., 2015), temporal action localization (Shou et al., 2016; Zhao et al., *Corresponding author. Sentence query: All four are once again talking in front of the camera. Single-step Interaction Ground Truth: [173.49s, 212.67s] misalignment Sentence query: All four are once again talking in front of the camera. Improved Inter- and Intra-attention Calibration calibration Improved Inter- and Intra-attention Calibration Improved Inter- and Intra-attention Calibration Figure 1: Illustration of our motivation. Upper: Previ- ous methods are mainly based on a single-step interac- tion with attention, which is insufficient to reason the complicated multi-modal relations, thus may lead to the misalignment on semantics. Bottom: we develop an iterative network with improved attention mecha- nism and calibration module, which can progressively align accurate semantic. 2017), and temporal sentence grounding (Gao et al., 2017; Anne Hendricks et al., 2017). Among them, temporal sentence grounding is the most challeng- ing task due to its complexity of multi-modal in- teractions and complicated context information. Given an untrimmed video, it aims to determine the segment boundaries including start and end times- tamps that contain …

Content truncated to 500 words