WSLLN:Weakly Supervised Natural Language Localization Networks

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 1481–1487, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 1481 WSLLN: Weakly Supervised Natural Language Localization Networks Mingfei Gao1 ∗ Larry S. Davis1 Richard Socher2 Caiming Xiong2 † 1University of Maryland 2Salesforce Research {mgao,lsd}@umiacs.umd.edu, {rsocher,cxiong}@salesforce.com Abstract We propose weakly supervised language lo- calization networks (WSLLN) to detect events in long, untrimmed videos given language queries. To learn the correspondence between visual segments and texts, most previous methods require temporal coordinates (start and end times) of events for training, which leads to high costs of annotation. WSLLN re- lieves the annotation burden by training with only video-sentence pairs without accessing to temporal locations of events. With a sim- ple end-to-end structure, WSLLN measures segment-text consistency and conducts seg- ment selection (conditioned on the text) simul- taneously. Results from both are merged and optimized as a video-sentence matching prob- lem. Experiments on ActivityNet Captions and DiDeMo demonstrate that WSLLN achieves state-of-the-art performance. 1 Introduction Extensive work has been done on temporal ac- tion/activity localization (Shou et al., 2016; Zhao et al., 2017; Dai et al., 2017; Buch et al., 2017; Gao et al., 2017c; Chao et al., 2018), where an ac- tion of interest is segmented from long, untrimmed videos. These methods only identify actions from a pre-defined set of categories, which limits their application to situations where only unconstrained language descriptions are available. This more general problem is referred to as natural language localization (NLL) (Hendricks et al., 2017; Gao et al., 2017a). The goal is to retrieve a temporal segment from an untrimmed video based on an ar- bitrary text query. Recent work focuses on learn- ing the mapping from visual segments to the input text (Hendricks et al., 2017; Gao et al., 2017a; Liu et al., 2018; Hendricks et al., 2018; Zhang et al., ∗Work done when the author was at Salesforce Research. †Corresponding author. 2018) and retrieving segments based on the align- ment scores. However, in order to successfully train a NLL model, a large number of diverse lan- guage descriptions are needed to describe differ- ent temporal segments of videos which incurs high human labeling cost. We propose Weakly Supervised Language Lo- calization Networks (WSLLN) which requires only video-sentence pairs during training with no information of where the activities temporally oc- cur. Intuitively, it is much easier to annotate video- level descriptions than segment-level descriptions. Moreover, when combined with text-based video retrieval techniques, video-sentence pairs may be obtained with minimum human intervention. The proposed model is simple and clean, and can be trained end-to-end in a single stage. We validate our model on ActivityNet Captions and DiDeMo. The results show that our model achieves the state- of-the-art of the weakly supervised approach and has comparable performance as some supervised approaches. 2 Related Work Temporal Action Localization in long videos is widely studied in both offline and online scenar- ios. In the …

Content truncated to 500 words