Weakly-Supervised Spatio-Temporally Grounding Natural Sentence in Video

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1884–1894 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 1884 Weakly-Supervised Spatio-Temporally Grounding Natural Sentence in Video Zhenfang Chen1∗ Lin Ma2† Wenhan Luo2† Kwan-Yee K. Wong1 1The University of Hong Kong 2Tencent AI Lab {zfchen, kykwong}@cs.hku.hk {forest.linma, whluo.china}@gmail.com Abstract In this paper, we address a novel task, namely weakly-supervised spatio-temporally ground- ing natural sentence in video. Specifically, given a natural sentence and a video, we local- ize a spatio-temporal tube in the video that se- mantically corresponds to the given sentence, with no reliance on any spatio-temporal anno- tations during training. First, a set of spatio- temporal tubes, referred to as instances, are extracted from the video. We then encode these instances and the sentence using our pro- posed attentive interactor which can exploit their fine-grained relationships to character- ize their matching behaviors. Besides a rank- ing loss, a novel diversity loss is introduced to train the proposed attentive interactor to strengthen the matching behaviors of reliable instance-sentence pairs and penalize the unre- liable ones. Moreover, we also contribute a dataset, called VID-sentence, based on the Im- ageNet video object detection dataset, to serve as a benchmark for our task. Extensive exper- imental results demonstrate the superiority of our model over the baseline approaches. Our code and the constructed VID-sentence dataset are available at: https://github.com/ JeffCHEN2017/WSSTG.git. 1 Introduction Given an image/video and a language query, im- age/video grounding aims to localize a spatial re- gion in the image (Plummer et al., 2015; Yu et al., 2017, 2018) or a specific frame in the video (Zhou et al., 2018) which semantically corresponds to the language query. Grounding has broad appli- cations, such as text based image retrieval (Chen et al., 2017; Ma et al., 2015), description genera- tion (Wang et al., 2018a; Rohrbach et al., 2017; ∗ Work done while Zhenfang Chen was a Research In- tern with Tencent AI Lab. † Corresponding authors. A brown and white dog is lying on the grass and then it stands up. ... ... Figure 1: The proposed WSSTG task aims to local- ize a spatio-temporal tube (i.e., the sequence of green bounding boxes) in the video which semantically cor- responds to the given sentence, with no reliance on any spatio-temporal annotations during training. Wang et al., 2018b), and question answer (Gao et al., 2018; Ma et al., 2016). Recently, promising progress has been made in image grounding (Yu et al., 2018; Chen et al., 2018c; Zhang et al., 2018) which heavily relies on fine-grained annotations in the form of region-sentence pairs. Fine-grained annotations for video grounding are more compli- cated and labor-intensive as one may need to an- notate a spatio-temporal tube (i.e., label the spatial region in each frame) in a video which semanti- cally corresponds to one language query. To avoid the intensive labor involved in dense annotations, (Huang et al., 2018) and (Zhou et al., 2018) considered the problem of weakly- supervised …

Content truncated to 500 words