Incorporating Visual Semantics into Sentence Representations within a Grounded Space

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 696–707, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 696 Incorporating Visual Semantics into Sentence Representations within a Grounded Space Patrick Bordes∗ 1 and ´Eloi Zablocki∗ 1 and Laure Soulier1 and Benjamin Piwowarski1 and Patrick Gallinari1,2 1 Sorbonne Universit´e, CNRS, LIP6, F-75005 Paris, France 2 Criteo AI Lab, Paris, France firstname.name@lip6.fr Abstract Language grounding is an active field aiming at enriching textual representations with visual information. Generally, textual and visual el- ements are embedded in the same representa- tion space, which implicitly assumes a one-to- one correspondence between modalities. This hypothesis does not hold when representing words, and becomes problematic when used to learn sentence representations — the focus of this paper — as a visual scene can be de- scribed by a wide variety of sentences. To overcome this limitation, we propose to trans- fer visual information to textual representa- tions by learning an intermediate representa- tion space: the grounded space. We further propose two new complementary objectives ensuring that (1) sentences associated with the same visual content are close in the grounded space and (2) similarities between related el- ements are preserved across modalities. We show that this model outperforms the previous state-of-the-art on classification and semantic relatedness tasks. 1 Introduction Representing text by vectors that capture mean- ingful semantics is a long-standing issue in Artifi- cial Intelligence. Distributional Semantic Models (Mikolov et al., 2013; Peters et al., 2018) are well- known recent efforts in this direction, based on the distributional hypothesis (Harris, 1954). They rely on large text corpora to learn word embeddings. At another granularity level, having high-quality and general-purpose sentence representations is crucial for all models that encode sentences into semantic vectors, such as the ones used in machine translation (Bahdanau et al., 2014) or relation ex- traction (Wang et al., 2019). Moreover, encod- ing semantics of sentences is paramount because ∗Equal contribution. sentences describe relationships between objects, and thus convey complex and high-level knowl- edge better than individual words (Norman, 1972). Relying only on text can lead to biased repre- sentations and unrealistic predictions such as “the sky is green” (Baroni, 2016). Besides, it has been shown that human understanding of language is grounded in physical reality and perceptual expe- rience (Fincher-Kiefer, 2001). To overcome this limitation, an emerging approach is to ground lan- guage in the visual world: this consists in lever- aging visual information, usually from images, to enrich textual representations.∗ Leveraging images resulted in improved lin- guistic representations on intrinsic and down- stream tasks (Bruni et al., 2014; Silberer and La- pata, 2014). In most of these approaches, cross- modal projections are learned to incorporate vi- sual semantics in the final representations (Lazari- dou et al., 2015b; Collell et al., 2017; Kiela et al., 2018). These works rely on paired textual and vi- sual data and the hypothesis of a one-to-one cor- respondence between modalities …

Content truncated to 500 words