Generating Image Descriptions Using Dependency Relational Patterns

Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 1250–1258, Uppsala, Sweden, 11-16 July 2010. c⃝2010 Association for Computational Linguistics Generating image descriptions using dependency relational patterns Ahmet Aker University of Sheffield a.aker@dcs.shef.ac.uk Robert Gaizauskas University of Sheffield r.gaizauskas@dcs.shef.ac.uk Abstract This paper presents a novel approach to automatic captioning of geo-tagged images by summarizing multiple web- documents that contain information re- lated to an image’s location. The summa- rizer is biased by dependency pattern mod- els towards sentences which contain fea- tures typically provided for different scene types such as those of churches, bridges, etc. Our results show that summaries bi- ased by dependency pattern models lead to significantly higher ROUGE scores than both n-gram language models reported in previous work and also Wikipedia base- line summaries. Summaries generated us- ing dependency patterns also lead to more readable summaries than those generated without dependency patterns. 1 Introduction The number of images tagged with location infor- mation on the web is growing rapidly, facilitated by the availability of GPS (Global Position Sys- tem) equipped cameras and phones, as well as by the widespread use of online social sites. The ma- jority of these images are indexed with GPS coor- dinates (latitude and longitude) only and/or have minimal captions. This typically small amount of textual information associated with the image is of limited usefulness for image indexing, organiza- tion and search. Therefore methods which could automatically supplement the information avail- able for image indexing and lead to improved im- age retrieval would be extremely useful. Following the general approach proposed by Aker and Gaizauskas (2009), in this paper we describe a method for automatic image caption- ing or caption enhancement starting with only a scene or subject type and a set of place names per- taining to an image – for example ⟨church, {St. Paul’s,London}⟩. Scene type and place names can be obtained automatically given GPS coordinates and compass information using techniques such as those described in Xin et al. (2010) – that task is not the focus of this paper. Our method applies only to images of static fea- tures of the built or natural landscape, i.e. objects with persistent geo-coordinates, such as buildings and mountains, and not to images of objects which move about in such landscapes, e.g. people, cars, clouds, etc. However, our technique is suitable not only for image captioning but in any application context that requires summary descriptions of in- stances of object classes, where the instance is to be characterized in terms of the features typically mentioned in describing members of the class. Aker and Gaizauskas (2009) have argued that humans appear to have a conceptual model of what is salient regarding a certain object type (e.g. church, bridge, etc.) and that this model informs their choice of what to say when describing an in- stance of this type. They also experimented with representing such conceptual models using n-gram language models derived from corpora consisting of collections of descriptions of instances of …

Content truncated to 500 words