Gaussian Visual-Linguistic Embedding for Zero-Shot Recognition

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 912–918, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Gaussian Visual-Linguistic Embedding for Zero-Shot Recognition Tanmoy Mukherjee and Timothy Hospedales Queen Mary University of London School of Electronic Engineering and Computer Science {k.m.tanmoy,t.hospedales}@qmul.ac.uk Abstract An exciting outcome of research at the inter- section of language and vision is that of zero- shot learning (ZSL). ZSL promises to scale visual recognition by borrowing distributed semantic models learned from linguistic cor- pora and turning them into visual recognition models. However the popular word-vector DSM embeddings are relatively impoverished in their expressivity as they model each word as a single vector point. In this paper we ex- plore word-distribution embeddings for ZSL. We present a visual-linguistic mapping for ZSL in the case where words and visual cat- egories are both represented by distributions. Experiments show improved results on ZSL benchmarks due to this better exploiting of intra-concept variability in each modality 1 Introduction Learning vector representations of word meaning is a topical area in computational linguistics. Based on the distributional hypothesis (Harris, 1954) – that words in similar context have similar meanings – distributed semantic models (DSM)s build vector representations based on corpus-extracted context. DSM approaches such as topic models (Blei et al., 2003), and more recently neural networks (Collobert et al., 2011; Mikolov et al., 2013) have had great success in a variety of lexical and semantic tasks (Arora et al., 2015; Schwenk, 2007). However despite their successes, classic DSMs are severely impoverished compared to humans due to learning solely from word cooccurrence without grounding in the outside world. This has motivated a wave of recent research into multi-modal and cross- modal learning that aims to ground DSMs in non- linguistic modalities (Bruni et al., 2014; Kiela and Bottou, 2014; Silberer and Lapata, 2014; ?). Such multi-modal DSMs are attractive because they learn richer representations than language-only models (e.g., that bananas are yellow fruit (Bruni et al., 2012b)), and thus often outperform language only models in various lexical tasks (Bruni et al., 2012a). In this paper, we focus on a key unique and prac- tically valuable capability enabled by cross-modal DSMs: that of zero-shot learning (ZSL). Zero-shot recognition aims to recognise visual categories in the absence of any training examples by cross-modal transfer from language. The idea is to use a lim- ited set of training data to learn a linguistic-visual mapping and then apply the induced function to map images from novel visual categories (unseen during training) to a linguistic embedding: thus enabling recognition in the absence of visual training exam- ples. ZSL has generated big impact (Lampert et al., 2009; Socher et al., 2013; Lazaridou et al., 2014) due to the potential of leveraging language to help visual recognition scale to many categories without labor intensive image annotation. DSMs typically generate vector embeddings of words, and hence ZSL is typically realised by vari- ants of vector-valued cross-modal regression. How- ever, such vector representations have limited …

Content truncated to 500 words