Midge: Generating Image Descriptions From Computer Vision Detections

Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 747–756, Avignon, France, April 23 - 27 2012. c⃝2012 Association for Computational Linguistics Midge: Generating Image Descriptions From Computer Vision Detections Margaret Mitchell† Xufeng Han§ Jesse Dodge‡‡ Alyssa Mensch∗∗ Amit Goyal†† Alex Berg§ Kota Yamaguchi§ Tamara Berg§ Karl Stratos∥ Hal Daum´e III†† † U. of Aberdeen and Oregon Health and Science University, m.mitchell@abdn.ac.uk § Stony Brook University, {aberg,tlberg,xufhan,kyamagu}@cs.stonybrook.edu ††U. of Maryland, {hal,amit}@umiacs.umd.edu ∥ Columbia University, stratos@cs.columbia.edu ‡‡U. of Washington, dodgejesse@gmail.com, ∗∗MIT, acmensch@mit.edu Abstract This paper introduces a novel generation system that composes humanlike descrip- tions of images from computer vision de- tections. By leveraging syntactically in- formed word co-occurrence statistics, the generator filters and constrains the noisy detections output from a vision system to generate syntactic trees that detail what the computer vision system sees. Results show that the generation system outper- forms state-of-the-art systems, automati- cally generating some of the most natural image descriptions to date. 1 Introduction It is becoming a real possibility for intelligent sys- tems to talk about the visual world. New ways of mapping computer vision to generated language have emerged in the past few years, with a fo- cus on pairing detections in an image to words (Farhadi et al., 2010; Li et al., 2011; Kulkarni et al., 2011; Yang et al., 2011). The goal in connect- ing vision to language has varied: systems have started producing language that is descriptive and poetic (Li et al., 2011), summaries that add con- tent where the computer vision system does not (Yang et al., 2011), and captions copied directly from other images that are globally (Farhadi et al., 2010) and locally similar (Ordonez et al., 2011). A commonality between all of these ap- proaches is that they aim to produce natural- sounding descriptions from computer vision de- tections. This commonality is our starting point: We aim to design a system capable of producing natural-sounding descriptions from computer vi- sion detections that are flexible enough to become more descriptive and poetic, or include likely in- The bus by the road with a clear blue sky Figure 1: Example image with generated description. formation from a language model, or to be short and simple, but as true to the image as possible. Rather than using a fixed template capable of generating one kind of utterance, our approach therefore lies in generating syntactic trees. We use a tree-generating process (Section 4.3) simi- lar to a Tree Substitution Grammar, but preserv- ing some of the idiosyncrasies of the Penn Tree- bank syntax (Marcus et al., 1995) on which most statistical parsers are developed. This allows us to automatically parse and train on an unlimited amount of text, creating data-driven models that flesh out descriptions around detected objects in a principled way, based on what is both likely and syntactically well-formed. An example generated description is given in Figure 1, and example vision output/natural lan- guage generation (NLG) input is given in Fig- ure 2. …

Content truncated to 500 words