Query-by-Example Image Retrieval using Visual Dependency Representations

Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 109–120, Dublin, Ireland, August 23-29 2014. Query-by-Example Image Retrieval using Visual Dependency Representations Desmond Elliott, Victor Lavrenko and Frank Keller Institute of Language, Communication, and Computation School of Informatics University of Edinburgh d.elliott@ed.ac.uk {vlavrenk,keller}@inf.ed.ac.uk Abstract Image retrieval models typically represent images as bags-of-terms, a representation that is well- suited to matching images based on the presence or absence of terms. For some information needs, such as searching for images of people performing actions, it may be useful to retain data about how parts of an image relate to each other. If the underlying representation of an image can distinguish between images where objects only co-occur from images where people are in- teracting with objects, then it should be possible to improve retrieval performance. In this paper we model the spatial relationships between image regions using Visual Dependency Represen- tations, a structured image representation that makes it possible to distinguish between object co-occurrence and interaction. In a query-by-example image retrieval experiment on data set of people performing actions, we find an 8.8% relative increase in MAP and an 8.6% relative increase in Precision@10 when images are represented using the Visual Dependency Represen- tation compared to a bag-of-terms baseline. 1 Introduction Every day millions of people search for images on the web, both professionally and for personal amuse- ment. The majority of image searches are aimed at finding a particular named entity, such as Justin Bieber or supernova, and a typical image retrieval system is well-suited to this type of information need because it represents an image as a bag-of-terms drawn from data surrounding the image, such as text, manual tags, and anchor text (Datta et al., 2008). It is not always possible to find useful terms in the sur- rounding data; the last decade has seen advances in automatic methods for assigning terms to images that have neither user-assigned tags, nor a textual description (Duygulu et al., 2002; Lavrenko et al., 2003; Guillaumin and Mensink, 2009). These automatic methods learn to associate the presence and absence of labels with the visual characteristics of an image, such as colour and texture distributions, shape, and points of interest, and can automatically generate a bag of terms for an unlabelled image. It is important to remember that not all information needs are entity-based: people also search for im- ages reflecting a mood, such as people having fun at a party, or an action, such as using a computer. The bag-of-terms representation is limited to matching images based on the presence or absence of terms, and not the relation of the terms to each other. Figures 1(a) and (b) highlight the problem with using unstructured representations for image retrieval: there is a person and a computer in both images but only (a) depicts a person actually using the computer. To address this problem with unstructured represen- tations we propose to represent the structure of an image using the Visual Dependency Representation …

Content truncated to 500 words