See No Evil, Say No Evil: Description Generation from Densely Labeled Images

Proceedings of the Third Joint Conference on Lexical and Computational Semantics (*SEM 2014), pages 110–120, Dublin, Ireland, August 23-24 2014. See No Evil, Say No Evil: Description Generation from Densely Labeled Images Mark Yatskar 1∗ my89@cs.washington.edu 1Computer Science & Engineering University of Washington Seattle, WA, 98195, USA Michel Galley2 mgalley@microsoft.com Lucy Vanderwende2 lucyv@microsoft.com Luke Zettlemoyer1 lsz@cs.washington.edu 2Microsoft Research One Microsoft Way Redmond, WA, 98052, USA Abstract This paper studies generation of descrip- tive sentences from densely annotated im- ages. Previous work studied generation from automatically detected visual infor- mation but produced a limited class of sen- tences, hindered by currently unreliable recognition of activities and attributes. In- stead, we collect human annotations of ob- jects, parts, attributes and activities in im- ages. These annotations allow us to build a significantly more comprehensive model of language generation and allow us to study what visual information is required to generate human-like descriptions. Ex- periments demonstrate high quality output and that activity annotations and relative spatial location of objects contribute most to producing high quality sentences. 1 Introduction Image descriptions compactly summarize com- plex visual scenes. For example, consider the de- scriptions of the image in Figure 1, which vary in content but focus on the women and what they are doing. Automatically generating such descriptions is challenging: a full system must understand the image, select the relevant visual content to present, and construct complete sentences. Existing sys- tems aim to address all of these challenges but use visual detectors for only a small vocabulary of words, typically nouns, associated with objects that can be reliably found.1 Such systems are blind ∗This work was conducted at Microsoft Research. 1While object recognition is improving (ImageNet accu- racy is over 90% for 1000 classes) progress in activity recog- nition has been slower; the state of the art is below 50% mean average precision for 40 activity classes (Yao et al., 2011). cars (Count:3) Isa: ride, vehicle,… Doing: parking,… Has: steering wheel,… Attrib: black, shiny,… children (Count:2) Isa: kids, children … Doing: biking, riding … Has: pants, bike … Attrib: young, small … bike (Count:1) Isa: bike, bicycle,… Doing: playing,… Has: chain, pedal,… Attrib: silver, white,… women(Count:3) Isa: girls, models,… Doing: smiling,... Has: shorts, bags,… Attrib: young, tan,… purses(Count:3) Isa: accessory,… Doing: containing,… Has: body, straps,… Attrib: black, soft,… sidewalk(Count:1) Isa: sidewalk, street,… Doing: laying,… Has: stone, cracks,… Attrib: flat, wide,… woman(Count:1) Isa: person, female,… Doing: pointing,… Has: nose, legs,… Attrib: tall, skinny,… tree(Count:1) Isa: plant,… Doing: growing,… Has: branches,… Attrib: tall, green,… kids(Count:5) Isa: group, teens,… Doing: walking,… Has: shoes, bags,… Attrib: young,… Five young people on the street, two sharing a bicycle. Several young people are walking near parked vehicles. Three girls with large handbags walking down the sidewalk. Three women walk down a city street, as seen from above. Three young woman walking down a sidewalk looking up. Figure 1: An annotated image with human generated sen- tence descriptions. Each bounding polygon encompasses one or more objects and is associated with a count …

Content truncated to 500 words