Image Description using Visual Dependency Representations

Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1292–1302, Seattle, Washington, USA, 18-21 October 2013. c⃝2013 Association for Computational Linguistics Image Description using Visual Dependency Representations Desmond Elliott School of Informatics University of Edinburgh d.elliott@ed.ac.uk Frank Keller School of Informatics University of Edinburgh keller@inf.ed.ac.uk Abstract Describing the main event of an image in- volves identifying the objects depicted and predicting the relationships between them. Previous approaches have represented images as unstructured bags of regions, which makes it difficult to accurately predict meaningful relationships between regions. In this pa- per, we introduce visual dependency represen- tations to capture the relationships between the objects in an image, and hypothesize that this representation can improve image de- scription. We test this hypothesis using a new data set of region-annotated images, as- sociated with visual dependency representa- tions and gold-standard descriptions. We de- scribe two template-based description gener- ation models that operate over visual depen- dency representations. In an image descrip- tion task, we find that these models outper- form approaches that rely on object proxim- ity or corpus information to generate descrip- tions on both automatic measures and on hu- man judgements. 1 Introduction Humans are readily able to produce a description of an image that correctly identifies the objects and ac- tions depicted. Automating this process is useful for applications such as image retrieval, where users can go beyond keyword-search to describe their infor- mation needs, caption generation for improving the accessibility of existing image collections, story il- lustration, and in assistive technology for blind and partially sighted people. Automatic image descrip- tion presents challenges on a number of levels: rec- ognizing the objects in an image and their attributes are difficult computer vision problems; while deter- mining how the objects interact, which relationships hold between them, and which events are depicted requires considerable background knowledge. Previous approaches to automatic description generation have typically tackled the problem us- ing an object recognition system in conjunction with a natural language generation component based on language models or templates (Kulkarni et al., 2011; Li et al., 2011). Some approaches have utilised the visual attributes of objects (Farhadi et al., 2010), generated descriptions by retrieving the descriptions of similar images (Ordonez et al., 2011; Kuznetsova et al., 2012), relied on an external corpus to pre- dict the relationships between objects (Yang et al., 2011), or combined sentence fragments using a tree- substitution grammar (Mitchell et al., 2012). A common aspect of existing work is that an im- age is represented as a bag of image regions. Bags of regions encode which objects co-occur in an im- age, but they are unable to express how the regions relate to each other, which makes it hard to describe what is happening. As an example, consider Fig- ure 1a, which depicts a man riding a bike. If the man was instead repairing the bike, then the bag- of-regions representation would be the same, even though the image would depict a different action and …

Content truncated to 500 words