Cross-Caption Coreference Resolution for Automatic Image Understanding

Proceedings of the Fourteenth Conference on Computational Natural Language Learning, pages 162–171, Uppsala, Sweden, 15-16 July 2010. c⃝2010 Association for Computational Linguistics Cross-caption coreference resolution for automatic image understanding Micah Hodosh Peter Young Cyrus Rashtchian Department of Computer ScienceJulia Hockenmaier University of Illinois at Urbana-Champaign {mhodosh2, pyoung2, crashtc2, juliahmr}@illinois.edu Abstract Recent work in computer vision has aimed to associate image regions with keywords describing the depicted entities, but ac- tual image ‘understanding’ would also re- quire identifying their attributes, relations and activities. Since this information can- not be conveyed by simple keywords, we have collected a corpus of “action” photos each associated with five descriptive cap- tions. In order to obtain a consistent se- mantic representation for each image, we need to first identify which NPs refer to the same entities. We present three hierar- chical Bayesian models for cross-caption coreference resolution. We have also cre- ated a simple ontology of entity classes that appear in images and evaluate how well these can be recovered. 1 Introduction Many photos capture a moment in time, telling a brief story of people, animals and objects, their at- tributes, and their relationship to each other. Al- though different people may give different inter- pretations to the same picture, people can read- ily interpret photos and describe the entities and events they perceive in complex sentences. This level of image understanding still remains an elu- sive goal for computer vision: although current methods may be able to identify the overall scene (Quattoni and Torralba, 2009) or some specific classes of entities (Felzenszwalb et al., 2008), they are only starting to be able to identify attributes of entities (Farhadi et al., 2009), and are far from recovering a complete semantic interpretation of the depicted situation. Like natural language pro- cessing, computer vision requires suitable training data, and there are currently no publicly available data sets that would enable the development of such systems. Photo sharing sites such as Flickr allow users to annotate images with keywords and other de- scriptions, and vision researchers have access to large collections of images annotated with key- words (e.g. the Corel collection). A lot of recent work in computer vision has been aimed at pre- dicting these keywords (Blei et al., 2003; Barnard et al., 2003; Feng and Lapata, 2008; Deschacht and Moens, 2007; Jeon et al., 2003). But key- words alone are not expressive enough to capture relations between entities. Some research has used the text that surrounds an image in a news arti- cle as a proxy (Feng and Lapata, 2008; Deschacht and Moens, 2007). However, in many cases, the surrounding text or a user-provided caption does not simply describe what is depicted in the image (since this is usually obvious to the human reader for which this text is intended), but provides ad- ditional information. We have collected a corpus of 8108 images associated with several simple de- scriptive captions. In contrast to the text near an image on the web, the captions in our corpus …

Content truncated to 500 words