Topic Models for Image Annotation and Text Illustration

Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the ACL, pages 831–839, Los Angeles, California, June 2010. c⃝2010 Association for Computational Linguistics Topic Models for Image Annotation and Text Illustration Yansong Feng and Mirella Lapata School of Informatics, University of Edinburgh 10 Crichton Street, Edinburgh EH8 9AB, UK Y.Feng-4@sms.ed.ac.uk, mlap@inf.ed.ac.uk Abstract Image annotation, the task of automatically generating description words for a picture, is a key component in various image search and retrieval applications. Creating image databases for model development is, however, costly and time consuming, since the key- words must be hand-coded and the process repeated for new collections. In this work we exploit the vast resource of images and documents available on the web for develop- ing image annotation models without any hu- man involvement. We describe a probabilistic model based on the assumption that images and their co-occurring textual data are gener- ated by mixtures of latent topics. We show that this model outperforms previously proposed approaches when applied to image annotation and the related task of text illustration despite the noisy nature of our dataset. 1 Introduction Recent years have witnessed the rapid growth of im- age collections available for searching and browsing over the Internet. Although image search engines are still in their infancy, initial research suggests that the deployed algorithms are not very accurate (Hawking et al., 1999). Given a query, search engines retrieve relevant pictures by analyzing the image caption (if it exists), textual descriptions found adjacent to the image, and other text-related factors such as the file name of the image. However, since they do not an- alyze the actual content of the images, search en- gines cannot be used to retrieve pictures from unan- notated collections. The ability to perform the an- notation task automatically would be of significant practical import for many image-based applications. Besides search and retrieval, other examples include browsing support (e.g., by clustering images into groups that are visually and semantically coherent) and story picturing (i.e., automatically suggesting images to illustrate text). Automatic image annotation is a popular task in computer vision; a large number of approaches have been proposed in the literature under many distinct learning paradigms. These range from supervised classification (Smeulders et al., 2000; Vailaya et al., 2001) to instantiations of the noisy-channel model (Duygulu et al., 2002), to clustering (Barnard et al., 2002), and methods inspired by information retrieval (Feng et al., 2004; Lavrenko et al., 2003). Despite differences in application and formulation, all these methods essentially attempt to learn the correlation between image features and words from examples of annotated images. The Corel database has been extensively used as a testbed for the development and evaluation of image annotation models. It is a collection of stock photographs, divided into themes (e.g., tigers, sunsets) each of which are associated with keywords (e.g., sun, sea) that are considered appropriate descriptors for all images belonging to the same theme. Unfortunately, the Corel database is not represen- tative of the size …

Content truncated to 500 words