Integrating Language and Vision to Generate Natural Language Descriptions of Videos in the Wild

Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 1218–1227, Dublin, Ireland, August 23-29 2014. Integrating Language and Vision to Generate Natural Language Descriptions of Videos in the Wild Jesse Thomason∗ University of Texas at Austin jesse@cs.utexas.edu Subhashini Venugopalan∗ University of Texas at Austin vsub@cs.utexas.edu Sergio Guadarrama University of California Berkeley sguada@eecs.berkeley.edu Kate Saenko University of Massachusetts Lowell saenko@cs.uml.edu Raymond Mooney University of Texas at Austin mooney@cs.utexas.edu Abstract This paper integrates techniques in natural language processing and computer vision to improve recognition and description of entities and activities in real-world videos. We propose a strategy for generating textual descriptions of videos by using a factor graph to combine visual detections with language statistics. We use state-of-the-art visual recognition systems to obtain confidences on entities, activities, and scenes present in the video. Our factor graph model combines these detection confidences with probabilistic knowledge mined from text corpora to estimate the most likely subject, verb, object, and place. Results on YouTube videos show that our approach im- proves both the joint detection of these latent, diverse sentence components and the detection of some individual components when compared to using the vision system alone, as well as over a previous n-gram language-modeling approach. The joint detection allows us to automatically generate more accurate, richer sentential descriptions of videos with a wide array of possible content. 1 Introduction Integrating language and vision is a topic that is attracting increasing attention in computational lin- guistics (Berg and Hockenmaier, 2013). Although there is a fair bit of research on generating natural- language descriptions of images (Feng and Lapata, 2013; Yang et al., 2011; Li et al., 2011; Ordonez et al., 2011), there is significantly less work on describing videos (Barbu et al., 2012; Guadarrama et al., 2013; Das et al., 2013; Rohrbach et al., 2013; Senina et al., 2014). In particular, much of the research on videos utilizes artificially constructed videos with prescribed sets of objects and actions (Barbu et al., 2012; Yu and Siskind, 2013). Generating natural-language descriptions of videos in the wild, such as those posted on YouTube, is a very challenging task. In this paper, we focus on selecting content for generating sentences to describe videos. Due to the large numbers of video actions and objects and scarcity of training data, we introduce a graphical model for integrating statistical linguistic knowledge mined from large text corpora with noisy computer vi- sion detections. This integration allows us to infer which vision detections to trust given prior linguistic knowledge. Using a large, realistic collection of YouTube videos, we demonstrate that this model effec- tively exploits linguistic knowledge to improve visual interpretation, producing more accurate descrip- tions compared to relying solely on visual information. For example, consider the frames of the video in Figure 1. Instead of generating the inaccurate description “A person is playing on the keyboard in the kitchen” using purely visual information, our system generates the more correct “A person is playing the piano in the house” by …

Content truncated to 500 words