Learning Visually Grounded Sentence Representations

Proceedings of NAACL-HLT 2018, pages 408–418 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics Learning Visually Grounded Sentence Representations Douwe Kiela Facebook AI Research dkiela@fb.com Alexis Conneau Facebook AI Research aconneau@fb.com Allan Jabri1 UC Berkeley ajabri@berkeley.edu Maximilian Nickel Facebook AI Research maxn@fb.com Abstract We investigate grounded sentence representa- tions, where we train a sentence encoder to predict the image features of a given caption— i.e., we try to “imagine” how a sentence would be depicted visually—and use the resultant fea- tures as sentence representations. We examine the quality of the learned representations on a variety of standard sentence representation quality benchmarks, showing improved perfor- mance for grounded models over non-grounded ones. In addition, we thoroughly analyze the extent to which grounding contributes to im- proved performance, and show that the system also learns improved word embeddings. 1 Introduction Following the word embedding upheaval of the past few years, one of NLP’s next big challenges has become the hunt for universal sentence rep- resentations: generic representations of sentence meaning that can be “plugged into” any kind of sys- tem or pipeline. Examples include Paragraph2Vec (Le and Mikolov, 2014), C-Phrase (Pham et al., 2015), SkipThought (Kiros et al., 2015) and Fast- Sent (Hill et al., 2016a). These representations tend to be learned from large corpora in an unsu- pervised setting, much like word embeddings, and effectively “transferred” to the task at hand. Purely text-based semantic models, which rep- resent word meaning as a distribution over other words (Harris, 1954; Turney and Pantel, 2010; Clark, 2015), suffer from the grounding problem (Harnad, 1990). It has been shown that ground- ing leads to improved performance on a variety of word-level tasks (Baroni, 2016; Kiela, 2017). Unsupervised sentence representation models are often doubly exposed to the grounding prob- lem, especially if they represent sentence mean- 1Work done while at Facebook AI Research. ings as a distribution over other sentences, as in SkipThought (Kiros et al., 2015). Here, we examine whether grounding also leads to improved sentence representations. In short, the grounding problem is characterized by the lack of an association between symbols and external infor- mation. We address this problem by aligning text with paired visual data and hypothesize that sen- tence representations can be enriched with external information—i.e., grounded—by forcing them to capture visual semantics. We investigate the per- formance of these representations and the effect of grounding on a variety of semantic benchmarks. There has been much recent interest in gener- ating actual images from text (Goodfellow et al., 2014; van den Oord et al., 2016; Mansimov et al., 2016). Our method takes a slightly different ap- proach: instead of predicting actual images, we train a deep recurrent neural network to predict the latent feature representation of images. That is, we are specifically interested in the semantic content of visual representations and how useful that information is for learning sentence represen- tations. One can think of this as trying to imag- ine, or form a “mental picture”, of …

Content truncated to 500 words