Text Analysis for Automatic Image Annotation

Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 1000–1007, Prague, Czech Republic, June 2007. c⃝2007 Association for Computational Linguistics Text Analysis for Automatic Image Annotation Koen Deschacht and Marie-Francine Moens Interdisciplinary Centre for Law & IT Department of Computer Science Katholieke Universiteit Leuven Tiensestraat 41, 3000 Leuven, Belgium {koen.deschacht,marie-france.moens}@law.kuleuven.ac.be Abstract We present a novel approach to automati- cally annotate images using associated text. We detect and classify all entities (persons and objects) in the text after which we de- termine the salience (the importance of an entity in a text) and visualness (the extent to which an entity can be perceived visually) of these entities. We combine these mea- sures to compute the probability that an en- tity is present in the image. The suitability of our approach was successfully tested on 100 image-text pairs of Yahoo! News. 1 Introduction Our society deals with a growing bulk of un- structured information such as text, images and video, a situation witnessed in many domains (news, biomedical information, intelligence information, business documents, etc.). This growth comes along with the demand for more effective tools to search and summarize this information. Moreover, there is the need to mine information from texts and images when they contribute to decision making by gov- ernments, businesses and other institutions. The capability to accurately recognize content in these sources would largely contribute to improved index- ing, classification, filtering, mining and interroga- tion. Algorithms and techniques for the disclosure of information from the different media have been de- veloped for every medium independently during the last decennium, but only recently the interplay be- tween these different media has become a topic of interest. One of the possible applications is to help analysis in one medium by employing information from another medium. In this paper we study text that is associated with an image, such as for instance image captions, video transcripts or surrounding text in a web page. We develop techniques that extract information from these texts to help with the diffi- cult task of accurate object recognition in images. Although images and associated texts never contain precisely the same information, in many situations the associated text offers valuable information that helps to interpret the image. The central objective of the CLASS project1 is to develop advanced learning methods that allow ima- ges, video and associated text to be automatically analyzed and structured. In this paper we test the feasibility of automatically annotating images by us- ing textual information in near-parallel image-text pairs, in which most of the content of the image corresponds to content of the text and vice versa. We will focus on entities such as persons and ob- jects. We will hereby take into account the text’s dis- course structure and semantics, which allow a more fine-grained identification of what content might be present in the image, and will enrich our model with world knowledge that is not present in the text. We will first discuss the corpus on which …

Content truncated to 500 words