From Visual Attributes to Adjectives through Decompositional Distributional Semantics

From Visual Attributes to Adjectives through Decompositional Distributional Semantics Angeliki Lazaridou Georgiana Dinu∗ Adam Liska Marco Baroni Center for Mind/Brain Sciences University of Trento {angeliki.lazaridou|georgiana.dinu|adam.liska|marco.baroni}@unitn.it Abstract As automated image analysis progresses, there is increasing interest in richer linguistic an- notation of pictures, with attributes of ob- jects (e.g., furry, brown...) attracting most attention. By building on the recent “zero- shot learning” approach, and paying atten- tion to the linguistic nature of attributes as noun modifiers, and specifically adjectives, we show that it is possible to tag images with attribute-denoting adjectives even when no training data containing the relevant an- notation are available. Our approach relies on two key observations. First, objects can be seen as bundles of attributes, typically ex- pressed as adjectival modifiers (a dog is some- thing furry, brown, etc.), and thus a function trained to map visual representations of ob- jects to nominal labels can implicitly learn to map attributes to adjectives. Second, ob- jects and attributes come together in pictures (the same thing is a dog and it is brown). We can thus achieve better attribute (and ob- ject) label retrieval by treating images as “vi- sual phrases”, and decomposing their linguis- tic representation into an attribute-denoting adjective and an object-denoting noun. Our approach performs comparably to a method exploiting manual attribute annotation, it out- performs various competitive alternatives in both attribute and object annotation, and it au- tomatically constructs attribute-centric repre- sentations that significantly improve perfor- mance in supervised object recognition. ∗ Current affiliation: Thomas J. Watson Research Center, IBM, gdinu@us.ibm.com 1 Introduction As the quality of image analysis algorithms im- proves, there is increasing interest in annotating im- ages with linguistic descriptions ranging from sin- gle words describing the depicted objects and their properties (Farhadi et al., 2009; Lampert et al., 2009) to richer expressions such as full-fledged im- age captions (Kulkarni et al., 2011; Mitchell et al., 2012). This trend has generated wide interest in lin- guistic annotations beyond concrete nouns, with the role of adjectives in image descriptions receiving, in particular, much attention. Adjectives are of special interest because of their central role in so-called attribute-centric image rep- resentations. This framework views objects as bun- dles of properties, or attributes, commonly ex- pressed by adjectives (e.g., furry, brown), and uses the latter as features to learn higher-level, seman- tically richer representations of objects (Farhadi et al., 2009).1 Attribute-based methods achieve better generalization of object classifiers with less train- ing data (Lampert et al., 2009), while at the same time producing semantic representations of visual concepts that more accurately model human se- 1In this paper, we assume that, just like nouns are the lin- guistic counterpart of visual objects, visual attributes are ex- pressed by adjectives. An informal survey of the relevant litera- ture suggests that, when attributes have linguistic labels, they are indeed mostly expressed by adjectives. There are some attributes, such as parts, that are more naturally expressed by prepositional phrases (PPs: with a tail). Interestingly, Dinu and Baroni (2014) showed …

Content truncated to 500 words