Composing Simple Image Descriptions using Web-scale N-grams

Proceedings of the Fifteenth Conference on Computational Natural Language Learning, pages 220–228, Portland, Oregon, USA, 23–24 June 2011. c⃝2011 Association for Computational Linguistics Composing Simple Image Descriptions using Web-scale N-grams Siming Li, Girish Kulkarni, Tamara L Berg, Alexander C Berg, and Yejin Choi Department of Computer Science Stony Brook University NY 11794, USA {silli, gkulkarni, tlberg, aberg, ychoi}@cs.stonybrook.edu Abstract Studying natural language, and especially how people describe the world around them can help us better understand the visual world. In turn, it can also help us in the quest to generate natural language that describes this world in a human manner. We present a simple yet effec- tive approach to automatically compose im- age descriptions given computer vision based inputs and using web-scale n-grams. Unlike most previous work that summarizes or re- trieves pre-existing text relevant to an image, our method composes sentences entirely from scratch. Experimental results indicate that it is viable to generate simple textual descriptions that are pertinent to the specific content of an image, while permitting creativity in the de- scription – making for more human-like anno- tations than previous approaches. 1 Introduction Gaining a better understanding of natural language, and especially natural language associated with im- ages helps drive research in both computer vision and natural language processing (e.g., Barnard et al. (2003), Pastra et al. (2003), Feng and Lapata (2010b)). In this paper, we look at how to exploit the enormous amount of textual data electronically available today, web-scale n-gram data in particular, in a simple yet highly effective approach to com- pose image descriptions in natural language. Auto- matic generation of image descriptions differs from automatic image tagging (e.g., Leong et al. (2010)) in that we aim to generate complex phrases or sen- tences describing images rather than predicting in- dividual words. These natural language descriptions can be useful for a variety of applications, includ- ing image retrieval, automatic video surveillance, and providing image interpretations for visually im- paired people. Our work contrasts to most previous approaches in four key aspects: first, we compose fresh sen- tences from scratch, instead of retrieving (Farhadi et al. (2010)), or summarizing existing text fragments associated with an image (e.g., Aker and Gaizauskas (2010), Feng and Lapata (2010a)). Second, we aim to generate textual descriptions that are truthful to the specific content of the image, whereas related (but subtly different) work in automatic caption gen- eration creates news-worthy text (Feng and Lapata (2010a)) or encyclopedic text (Aker and Gaizauskas (2010)) that is contextually relevant to the image, but not closely pertinent to the specific content of the image. Third, we aim to build a general image de- scription method as compared to work that requires domain specific hand-written grammar rules (Yao et al. (2010)). Last, we allow for some creativity in the generation process which produces more human- like descriptions than a closely related, very recent approach that drives annotation more directly from computer vision inputs (Kulkarni et al., 2011). In this work, we propose a novel …

Content truncated to 500 words