Automatic Image Annotation Using Auxiliary Text Information

Proceedings of ACL-08: HLT, pages 272–280, Columbus, Ohio, USA, June 2008. c⃝2008 Association for Computational Linguistics Automatic Image Annotation Using Auxiliary Text Information Yansong Feng and Mirella Lapata School of Informatics, University of Edinburgh 2 Buccleuch Place, Edinburgh EH8 9LW, UK Y.Feng-4@sms.ed.ac.uk, mlap@inf.ed.ac.uk Abstract The availability of databases of images labeled with keywords is necessary for developing and evaluating image annotation models. Dataset collection is however a costly and time con- suming task. In this paper we exploit the vast resource of images available on the web. We create a database of pictures that are natu- rally embedded into news articles and propose to use their captions as a proxy for annota- tion keywords. Experimental results show that an image annotation model can be developed on this dataset alone without the overhead of manual annotation. We also demonstrate that the news article associated with the picture can be used to boost image annotation perfor- mance. 1 Introduction As the number of image collections is rapidly grow- ing, so does the need to browse and search them. Recent years have witnessed significant progress in developing methods for image retrieval1, many of which are query-based. Given a database of images, each annotated with keywords, the query is used to retrieve relevant pictures under the assumption that the annotations can essentially capture their seman- tics. One stumbling block to the widespread use of query-based image retrieval systems is obtaining the keywords for the images. Since manual annotation is expensive, time-consuming and practically infea- sible for large databases, there has been great in- 1The approaches are too numerous to list; we refer the inter- ested reader to Datta et al. (2005) for an overview. terest in automating the image annotation process (see references). More formally, given an image I with visual features Vi = {v1,v2,...,vN} and a set of keywords W = {w1,w2,...,wM}, the task con- sists in finding automatically the keyword subset WI ⊂ W, which can appropriately describe the im- age I. Indeed, several approaches have been pro- posed to solve this problem under a variety of learn- ing paradigms. These range from supervised clas- sification (Vailaya et al., 2001; Smeulders et al., 2000) to instantiations of the noisy-channel model (Duygulu et al., 2002), to clustering (Barnard et al., 2002), and methods inspired by information retrieval (Lavrenko et al., 2003; Feng et al., 2004). Obviously in order to develop accurate image an- notation models, some manually labeled data is re- quired. Previous approaches have been developed and tested almost exclusively on the Corel database. The latter contains 600 CD-ROMs, each contain- ing about 100 images representing the same topic or concept, e.g., people, landscape, male. Each topic is associated with keywords and these are assumed to also describe the images under this topic. As an example consider the pictures in Figure 1 which are classified under the topic male and have the descrip- tion keywords man, male, people, cloth, and face. Current image annotation methods work well when large amounts of labeled …

Content truncated to 500 words