Discriminative Unsupervised Alignment of Natural Language Instructions with Corresponding Video Segments

Human Language Technologies: The 2015 Annual Conference of the North American Chapter of the ACL, pages 164–174, Denver, Colorado, May 31 – June 5, 2015. c⃝2015 Association for Computational Linguistics Discriminative Unsupervised Alignment of Natural Language Instructions with Corresponding Video Segments Iftekhar Naim1, Young Chol Song1, Qiguang Liu1, Liang Huang2, Henry Kautz1, Jiebo Luo1 and Daniel Gildea1 1Department of Computer Science, University of Rochester, Rochester, NY 14627 2Queens College & Graduate Center, City University of New York, Flushing, NY 11367 Abstract We address the problem of automatically aligning natural language sentences with cor- responding video segments without any direct supervision. Most existing algorithms for in- tegrating language with videos rely on hand- aligned parallel data, where each natural lan- guage sentence is manually aligned with its corresponding image or video segment. Re- cently, fully unsupervised alignment of text with video has been shown to be feasible us- ing hierarchical generative models. In contrast to the previous generative models, we propose three latent-variable discriminative models for the unsupervised alignment task. The pro- posed discriminative models are capable of in- corporating domain knowledge, by adding di- verse and overlapping features. The results show that discriminative models outperform the generative models in terms of alignment accuracy. 1 Introduction Learning to integrate natural language descriptions with video events is attracting increasing attention in the natural language processing and computer vi- sion communities. The Grounded Language Learn- ing task aims to map the meaning of natural lan- guage expressions to their corresponding referents in videos (e.g., objects, actions, and events) with- out any dictionary. Most existing grounded lan- guage learning algorithms are either supervised or weakly-supervised. During the training stage, they assume each video is pre-segmented to chunks of short duration, and each video segment is manually Label the bottle Add 500 mL of DI water to the labeled bottle Transfer 1 mL of MgSO4 to the 50 mL Falcon tube Alignment of Video Segments with Text Sentences Add 500 mL of DI water to the labeled bottle Verb Nouns Video Objects Touched by Hands Figure 1: The proposed discriminative learning algorithm aligns protocol sentences to corresponding video frames. We incorporate features that can learn the co-occurrences of nouns and verbs in the sentences with the objects in the video. aligned with a natural language sentence that de- scribes that segment. Manually aligning each video segment with a sentence is tedious, especially for long videos. Therefore, it is desirable to automati- cally align video segments with their corresponding natural language sentences without direct supervi- sion. Recently, Naim et al. (2014) proposed an unsuper- vised learning algorithm for automatically aligning sentences in a document with corresponding video segments. Given a sequence of natural language instructions and an unaligned video recording of a person following these instructions, a hierarchical generative model was applied to align each instruc- 164 tion to its corresponding video segment, and to align nouns in each instruction to their corresponding ob- jects in the video. We extend this generative align- ment framework by …

Content truncated to 500 words