How Many Words Is a Picture Worth? Automatic Caption Generation for News Images

Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 1239–1249, Uppsala, Sweden, 11-16 July 2010. c⃝2010 Association for Computational Linguistics How Many Words is a Picture Worth? Automatic Caption Generation for News Images Yansong Feng and Mirella Lapata School of Informatics, University of Edinburgh 10 Crichton Street, Edinburgh EH8 9AB, UK Y.Feng-4@sms.ed.ac.uk, mlap@inf.ed.ac.uk Abstract In this paper we tackle the problem of au- tomatic caption generation for news im- ages. Our approach leverages the vast re- source of pictures available on the web and the fact that many of them are cap- tioned. Inspired by recent work in sum- marization, we propose extractive and ab- stractive caption generation models. They both operate over the output of a proba- bilistic image annotation model that pre- processes the pictures and suggests key- words to describe their content. Exper- imental results show that an abstractive model defined over phrases is superior to extractive methods. 1 Introduction Recent years have witnessed an unprecedented growth in the amount of digital information avail- able on the Internet. Flickr, one of the best known photo sharing websites, hosts more than three bil- lion images, with approximately 2.5 million im- ages being uploaded every day.1 Many on-line news sites like CNN, Yahoo!, and BBC publish images with their stories and even provide photo feeds related to current events. Browsing and find- ing pictures in large-scale and heterogeneous col- lections is an important problem that has attracted much interest within information retrieval. Many of the search engines deployed on the web retrieve images without analyzing their con- tent, simply by matching user queries against col- located textual information. Examples include meta-data (e.g., the image’s file name and for- mat), user-annotated tags, captions, and gener- ally text surrounding the image. As this limits the applicability of search engines (images that 1http://www.techcrunch.com/2008/11/03/ three-billion-photos-at-flickr/ do not coincide with textual data cannot be re- trieved), a great deal of work has focused on the development of methods that generate description words for a picture automatically. The literature is littered with various attempts to learn the as- sociations between image features and words us- ing supervised classification (Vailaya et al., 2001; Smeulders et al., 2000), instantiations of the noisy- channel model (Duygulu et al., 2002), latent vari- able models (Blei and Jordan, 2003; Barnard et al., 2002; Wang et al., 2009), and models inspired by information retrieval (Lavrenko et al., 2003; Feng et al., 2004). In this paper we go one step further and gen- erate captions for images rather than individual keywords. Although image indexing techniques based on keywords are popular and the method of choice for image retrieval engines, there are good reasons for using more linguistically meaningful descriptions. A list of keywords is often ambigu- ous. An image annotated with the words blue, sky, car could depict a blue car or a blue sky, whereas the caption “car running under the blue sky” would make the relations between the words explicit. Automatic caption generation could im- prove …

Content truncated to 500 words