Unsupervised Disambiguation of Image Captions

First Joint Conference on Lexical and Computational Semantics (*SEM), pages 85–89, Montr´eal, Canada, June 7-8, 2012. c⃝2012 Association for Computational Linguistics Unsupervised Disambiguation of Image Captions Wesley May, Sanja Fidler, Afsaneh Fazly, Sven Dickinson, and Suzanne Stevenson Department of Computer Science University of Toronto Toronto, Ontario, Canada, M5S 3G4 {wesley,fidler,afsaneh,sven,suzanne}@cs.toronto.edu Abstract Given a set of images with related captions, our goal is to show how visual features can improve the accuracy of unsupervised word sense disambiguation when the textual con- text is very small, as this sort of data is com- mon in news and social media. We extend previous work in unsupervised text-only dis- ambiguation with methods that integrate text and images. We construct a corpus by using Amazon Mechanical Turk to caption sense- tagged images gathered from ImageNet. Us- ing a Yarowsky-inspired algorithm, we show that gains can be made over text-only disam- biguation, as well as multimodal approaches such as Latent Dirichlet Allocation. 1 Introduction We examine the problem of performing unsuper- vised word sense disambiguation (WSD) in situa- tions with little text, but where additional informa- tion is available in the form of an image. Such situ- ations include captioned newswire photos, and pic- tures in social media where the textual context is of- ten no larger than a tweet. Unsupervised WSD has been shown to work very well when the target word is embedded in a large We thank NSERC and U. Toronto for financial support. Fi- dler and Dickinson were sponsored by the Army Research Lab- oratory and this research was accomplished in part under Co- operative Agreement Number W911NF-10-2-0060. The views and conclusions contained in this document are those of the au- thors and should not be interpreted as representing the official policies, either express or implied, of the Army Research Lab- oratory or the U.S. Government. Figure 1: “The crane was so massive it blocked the sun.” Which sense of crane? With images the answer is clear. quantity of text (Yarowsky, 1995). However, if the only available text is “The crane was so massive it blocked the sun” (see Fig. 1), then text-only dis- ambiguation becomes much more difficult; a human could do little more than guess. But if an image is available, the intended sense is much clearer. We develop an unsupervised WSD algorithm based on Yarowsky’s that uses words in a short caption along with “visual words” from the captioned image to choose the best of two possible senses of an ambigu- ous keyword describing the content of the image. Language-vision integration is a quickly develop- ing field, and a number of researchers have explored the possibility of combining text and visual features in various multimodal tasks. Leong and Mihal- cea (2011) explored semantic relatedness between words and images to better exploit multimodal con- tent. Jamieson et al. (2009) and Feng and Lap- ata (2010) combined text and vision to perform ef- fective image annotation. Barnard and colleagues (2003; 2005) showed that supervised WSD by could be improved with visual …

Content truncated to 500 words