Going Beyond Text: A Hybrid Image-Text Approach for Measuring Word Relatedness

Proceedings of the 5th International Joint Conference on Natural Language Processing, pages 1403–1407, Chiang Mai, Thailand, November 8 – 13, 2011. c⃝2011 AFNLP Going Beyond Text: A Hybrid Image-Text Approach for Measuring Word Relatedness Chee Wee Leong and Rada Mihalcea Department of Computer Science and Engineering University of North Texas cheeweeleong@my.unt.edu, rada@cs.unt.edu Abstract Traditional approaches to semantic relat- edness are often restricted to text-based methods, which typically disregard other multimodal knowledge sources. In this paper, we propose a novel image-based metric to estimate the relatedness of words, and demonstrate the promise of this method through comparative evalua- tions on three standard datasets. We also show that a hybrid image-text approach can lead to improvements in word related- ness, confirming the applicability of visual cues as a possible orthogonal information source. 1 Introduction Measuring the semantic relatedness of words is an important task with applications in information extraction and retrieval, query reformulation, word sense disambiguation, plagiarism detection and textual entailment. Owing mainly to the nature of this task, research efforts in the past have typically centered around methodologies employing the use of knowledge-based or corpus-based textual re- sources, with only little (if any) work paying at- tention to evidence provided by other multimodal sources, such as visual cues presented by the im- ages that are associated with a given word. While it can be shown that the human cognitive system is sensitive to visual information, and incorporating a dual linguistic-and-pictorial representation of in- formation can actually enhance knowledge acqui- sition (Potter and Faulconer, 1975), the use of vi- sual information to improve tasks in natural lan- guage processing has been largely unexplored. In this paper, we hypothesize that the related- ness between the visual representations of a pair of words can be effectively used to gauge their similarity. We first discuss a technique widely used in computer vision termed as “bag of visual words” to show how distinctive features of an im- age can be harvested. We next introduce the main resource, ImageNet, used in our work to bridge the semantic gap between words and images. Finally, we show how a new relatedness metric based ex- clusively on visual information can be constructed for the semantic relatedness task. We evaluate this metric alongside existing corpus-based (Tur- ney and Pantel, 2010) and knowledge-based met- rics (Pedersen et al., 2004) either in a standalone or combined setting and present our findings. 2 Bag of Visual Words Inspired by the bag-of-words approach employed in information retrieval, the “bag of visual code- words” is a similar technique used mainly for scene classification (Yang et al., 2007). Starting with an image collection, visual features are first extracted as data points from each image. By pro- jecting data points from all the images into a com- mon space and grouping them into a large num- ber of clusters such that similar data points are assigned to the same cluster, we can treat each cluster as a “visual codeword” and express every image in the collection as a …

Content truncated to 500 words