Corpus-Guided Sentence Generation of Natural Images

Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 444–454, Edinburgh, Scotland, UK, July 27–31, 2011. c⃝2011 Association for Computational Linguistics Corpus-Guided Sentence Generation of Natural Images Yezhou Yang † and Ching Lik Teo † and Hal Daum´e III and Yiannis Aloimonos University of Maryland Institute for Advanced Computer Studies College Park, Maryland 20742, USA {yzyang, cteo, hal, yiannis}@umiacs.umd.edu Abstract We propose a sentence generation strategy that describes images by predicting the most likely nouns, verbs, scenes and prepositions that make up the core sentence structure. The input are initial noisy estimates of the objects and scenes detected in the image using state of the art trained detectors. As predicting actions from still images directly is unreliable, we use a language model trained from the English Gi- gaword corpus to obtain their estimates; to- gether with probabilities of co-located nouns, scenes and prepositions. We use these esti- mates as parameters on a HMM that models the sentence generation process, with hidden nodes as sentence components and image de- tections as the emissions. Experimental re- sults show that our strategy of combining vi- sion and language produces readable and de- scriptive sentences compared to naive strate- gies that use vision alone. 1 Introduction What happens when you see a picture? The most natural thing would be to describe it using words: using speech or text. This description of an image is the output of an extremely complex process that in- volves: 1) perception in the Visual space, 2) ground- ing to World Knowledge in the Language Space and 3) speech/text production (see Fig. 1). Each of these components are challenging in their own right and are still considered open problems in the vision and linguistics fields. In this paper, we introduce a com- putational framework that attempts to integrate these †indicates equal contribution. Figure 1: The processes involved for describing a scene. components together. Our hypothesis is based on the assumption that natural images accurately reflect common everyday scenarios which are captured in language. For example, knowing that boats usually occur over water will enable us to constrain the possible scenes a boat can occur and exclude highly unlikely ones – street, highway. It also en- ables us to predict likely actions (Verbs) given the current object detections in the image: detecting a dog with a person will likely induce walk rather than swim, jump, fly. Key to our approach is the use of a large generic corpus such as the English Gigaword [Graff, 2003] as the semantic grounding to predict and correct the initial and often noisy vi- sual detections of an image to produce a reasonable sentence that succinctly describes the image. In order to get an idea of the difficulty of this task, it is important to first define what makes up 444 Figure 2: Illustration of various perceptual challenges for sentence generation for images. (a) Different images with semantically the same content. (b) Pose relates ambigu- ously to actions in real images. See …

Content truncated to 500 words