MCSE: Multimodal Contrastive Learning of Sentence Embeddings

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5959 - 5969 July 10-15, 2022 ©2022 Association for Computational Linguistics MCSE: Multimodal Contrastive Learning of Sentence Embeddings Miaoran Zhang, Marius Mosbach, David Ifeoluwa Adelani, Michael A. Hedderich, and Dietrich Klakow Spoken Language Systems (LSV) Saarland Informatics Campus, Saarland University, Germany {mzhang,mmosbach,didelani}@lsv.uni-saarland.de {mhedderich,dklakow}@lsv.uni-saarland.de Abstract Learning semantically meaningful sentence em- beddings is an open problem in natural lan- guage processing. In this work, we propose a sentence embedding learning approach that ex- ploits both visual and textual information via a multimodal contrastive objective. Through ex- periments on a variety of semantic textual simi- larity tasks, we demonstrate that our approach consistently improves the performance across various datasets and pre-trained encoders. In particular, combining a small amount of multi- modal data with a large text-only corpus, we im- prove the state-of-the-art average Spearman’s correlation by 1.7%. By analyzing the proper- ties of the textual embedding space, we show that our model excels in aligning semantically similar sentences, providing an explanation for its improved performance. 1 Introduction Sentence embedding learning, i.e., encoding sen- tences into fixed-length vectors that faithfully re- flect the semantic relatedness among sentences, is a fundamental challenge in natural language pro- cessing (NLP). Despite the tremendous success of pre-trained language models (PLMs), such as BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019), it has been shown that the off-the- shelf sentence embeddings of PLMs without fine- tuning are even inferior to averaging Glove embed- dings (Pennington et al., 2014) in terms of semantic similarity measure (Reimers and Gurevych, 2019). Hence, recent research (Li et al., 2020; Zhang et al., 2020; Su et al., 2021) focuses on adjusting the orig- inal sentence embeddings derived from PLMs in an unsupervised manner. In particular, there has been growing interest in adopting contrastive learn- ing objectives to achieve this goal (Carlsson et al., 2020; Kim et al., 2021; Gao et al., 2021). Although purely text-based models have led to impressive progress, it remains an open question to what extent they capture the deeper notion of sen- tence meaning beyond the statistical distribution of texts, which lies outside of the text and is grounded in the real-world (Bender and Koller, 2020; Bisk et al., 2020). As a central part of the human per- ceptual experience, vision has been shown to be effective in grounding language models and im- proving performance on various NLP tasks (Zhang et al., 2019; Bordes et al., 2019; Zhao and Titov, 2020). We hypothesize that using vision as supple- mentary semantic information can further promote sentence representation learning. In this work, we propose MCSE, an approach for multimodal contrastive learning of sentence embeddings. To exploit both visual and textual in- formation, we adopt the state-of-the-art contrastive sentence embedding framework SimCSE (Gao et al., 2021) and extend it with a multimodal con- trastive objective. In addition to the textual objec- tive in SimCSE that maximizes agreement between positive sentence …

Content truncated to 500 words