From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions

From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions Peter Young Alice Lai Micah Hodosh Julia Hockenmaier Department of Computer Science University of Illinois at Urbana-Champaign {pyoung2, aylai2, mhodosh2, juliahmr}@illinois.edu Abstract We propose to use the visual denotations of linguistic expressions (i.e. the set of images they describe) to define novel denotational similarity metrics, which we show to be at least as beneficial as distributional similarities for two tasks that require semantic inference. To compute these denotational similarities, we construct a denotation graph, i.e. a subsump- tion hierarchy over constituents and their de- notations, based on a large corpus of 30K im- ages and 150K descriptive captions. 1 Introduction The ability to draw inferences from text is a prereq- uisite for language understanding. These inferences are what makes it possible for even brief descrip- tions of everyday scenes to evoke rich mental im- ages. For example, we would expect an image of people shopping in a supermarket to depict aisles of produce or other goods, and we would expect most of these people to be customers who are either standing or walking around. But such inferences require a great deal of commonsense world knowl- edge. Standard distributional approaches to lexical similarity (Section 2.1) are very effective at iden- tifying which words are related to the same topic, and can provide useful features for systems that per- form semantic inferences (Mirkin et al., 2009), but are not suited to capture precise entailments between complex expressions. In this paper, we propose a novel approach for the automatic acquisition of de- notational similarities between descriptions of ev- eryday situations (Section 2). We define the (visual) denotation of a linguistic expression as the set of im- ages it describes. We create a corpus of images of everyday activities (each paired with multiple cap- tions; Section 3) to construct a large scale visual de- notation graph which associates image descriptions with their denotations (Section 4). The algorithm that constructs the denotation graph uses purely syn- tactic and lexical rules to produce simpler captions (which have a larger denotation). But since each image is originally associated with several captions, the graph can also capture similarities between syn- tactically and lexically unrelated descriptions. We apply these similarities to two different tasks (Sec- tions 6 and 7): an approximate entailment recogni- tion task for our domain, where the goal is to decide whether the hypothesis (a brief image caption) refers to the same image as the premises (four longer cap- tions), and the recently introduced Semantic Textual Similarity task (Agirre et al., 2012), which can be viewed as a graded (rather than binary) version of paraphrase detection. Both tasks require semantic inference, and our results indicate that denotational similarities are at least as effective as standard ap- proaches to similarity. Our code and data set, as well as the denotation graph itself and the lexical similarities we define over it are available for re- search purposes at http://nlp.cs.illinois.edu/ Denotation.html. 2 Towards Denotational …

Content truncated to 500 words