Grounded Compositional Semantics for Finding and Describing Images with Sentences

Grounded Compositional Semantics for Finding and Describing Images with Sentences Richard Socher, Andrej Karpathy, Quoc V. Le*, Christopher D. Manning, Andrew Y. Ng Stanford University, Computer Science Department, *Google Inc. richard@socher.org, karpathy@cs.stanford.edu, qvl@google.com, manning@stanford.edu, ang@cs.stanford.edu Abstract Previous work on Recursive Neural Networks (RNNs) shows that these models can produce compositional feature vectors for accurately representing and classifying sentences or im- ages. However, the sentence vectors of previ- ous models cannot accurately represent visu- ally grounded meaning. We introduce the DT- RNN model which uses dependency trees to embed sentences into a vector space in order to retrieve images that are described by those sentences. Unlike previous RNN-based mod- els which use constituency trees, DT-RNNs naturally focus on the action and agents in a sentence. They are better able to abstract from the details of word order and syntactic expression. DT-RNNs outperform other re- cursive and recurrent neural networks, kernel- ized CCA and a bag-of-words baseline on the tasks of finding an image that fits a sentence description and vice versa. They also give more similar representations to sentences that describe the same image. 1 Introduction Single word vector spaces are widely used (Turney and Pantel, 2010) and successful at classifying sin- gle words and capturing their meaning (Collobert and Weston, 2008; Huang et al., 2012; Mikolov et al., 2013). Since words rarely appear in isolation, the task of learning compositional meaning repre- sentations for longer phrases has recently received a lot of attention (Mitchell and Lapata, 2010; Socher et al., 2010; Socher et al., 2012; Grefenstette et al., 2013). Similarly, classifying whole images into a fixed set of classes also achieves very high perfor- mance (Le et al., 2012; Krizhevsky et al., 2012). However, similar to words, objects in images are of- ten seen in relationships with other objects which are not adequately described by a single label. In this work, we introduce a model, illustrated in Fig. 1, which learns to map sentences and images into a common embedding space in order to be able to retrieve one from the other. We assume word and image representations are first learned in their re- spective single modalities but finally mapped into a jointly learned multimodal embedding space. Our model for mapping sentences into this space is based on ideas from Recursive Neural Networks (RNNs) (Pollack, 1990; Costa et al., 2003; Socher et al., 2011b). However, unlike all previous RNN models which are based on constituency trees (CT- RNNs), our model computes compositional vector representations inside dependency trees. The com- positional vectors computed by this new dependency tree RNN (DT-RNN) capture more of the meaning of sentences, where we define meaning in terms of similarity to a “visual representation” of the textual description. DT-RNN induced vector representa- tions of sentences are more robust to changes in the syntactic structure or word order than related mod- els such as CT-RNNs or Recurrent Neural Networks since they naturally focus on a sentence’s action and its agents. We evaluate and compare DT-RNN induced rep- …

Content truncated to 500 words