Summary Generation for Toponym-referenced Images using Object Type Language Models

International Conference RANLP 2009 - Borovets, Bulgaria, pages 6–11 Summary Generation for Toponym-Referenced Images using Object Type Language Models Ahmet Aker & Robert Gaizauskas Department of Computer Science University of Sheffield Sheffield, S1 4DP, UK A.Aker, R.Gaizauskas@dcs.shef.ac.uk Abstract This paper presents a novel approach to au- tomatic captioning of toponym-referenced im- ages. The automatic captioning procedure works by summarizing multiple web-documents that contain information related to an image’s lo- cation. Our summarizer can generate both query-based and language model-biased multi- document summaries. The models are created from large numbers of existing articles pertaining to places of the same “object type”. Evaluation relative to human written captions shows that when language models are used to bias the sum- marizer the summaries score more highly than the non-biased ones. Keywords Multi-Document Summarization, Image Captioning, Lan- 1guage Models, Statistical Methods, NLP Introduction In recent years the number of images on the web has grown immensely, facilitated by the development of cheap digital hardware and the availability of online image sharing social sites. Many of these images are tagged only with place names or contain minimal cap- tions that include locational information. This small amount of textual information associated with the im- age is of limited usefulness for image indexing, organi- zation and search. What would be useful is a means to generate or augment captions automatically based on existing data. Attempts towards automatic generation of image captions have been previously reported. Deschacht & Moens [6] and Mori et al. [14] generate image captions automatically by analyzing image-related text from the immediate context of the image, e.g. the surround- ing text in HTML documents. The authors identify named entities and other noun phrases in the image- related text and assign these to the image as captions. Other approaches create image captions by taking into consideration image features (colour, shape and tex- ture) as well as image-related text [22, 14, 4, 7, 3, 15, 8]. These approaches analyze only the immediate textual context of the image. However, generating image cap- tions based on the immediate context of the image can result in an image description which does not describe the image at all. Marsch & White [13] argue that the content of an image and its immediate text have little semantic agreement and this can, according to Purves et al. [16], be misleading to image retrieval. Furthermore, these approaches assume that the image has been obtained from a document. In cases where there is no document associated with the image, which is the scenario we are principally concerned with, these techniques are not applicable. In this paper, we propose a technique for auto- matic image captioning or caption enhancement start- ing with only a set of place names pertaining to an image. The technique applies just to images of static features of the built or natural landscape (e.g. build- ings, mountains, etc.) and not to images of objects which move about in such landscapes (e.g. people, cars, clouds, etc.). Our approach is based on extractive …

Content truncated to 500 words