Contrastive Visual Semantic Pretraining Magnifies the Semantics of Natural Language Representations

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 3050 - 3061 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Contrastive Visual Semantic Pretraining Magnifies the Semantics of Natural Language Representations Robert Wolfe University of Washington rwolfe3@uw.edu Aylin Caliskan University of Washington aylin@uw.edu Abstract We examine the effects of contrastive visual semantic pretraining by comparing the geom- etry and semantic properties of contextualized English language representations formed by GPT-2 and CLIP, a zero-shot multimodal im- age classifier which adapts the GPT-2 architec- ture to encode image captions. We find that contrastive visual semantic pretraining signifi- cantly mitigates the anisotropy found in contex- tualized word embeddings from GPT-2, such that the intra-layer self-similarity (mean pair- wise cosine similarity) of CLIP word embed- dings is under .25 in all layers, compared to greater than .95 in the top layer of GPT-2. CLIP word embeddings outperform GPT-2 on word- level semantic intrinsic evaluation tasks, and achieve a new corpus-based state of the art for the RG65 evaluation, at .88. CLIP also forms fine-grained semantic representations of sen- tences, and obtains Spearman’s ρ = .73 on the SemEval-2017 Semantic Textual Similarity Benchmark with no fine-tuning, compared to no greater than ρ = .45 in any layer of GPT-2. Finally, intra-layer self-similarity of CLIP sen- tence embeddings decreases as the layer index increases, finishing at .25 in the top layer, while the self-similarity of GPT-2 sentence embed- dings formed using the EOS token increases layer-over-layer and never falls below .97. Our results indicate that high anisotropy is not an in- evitable consequence of contextualization, and that visual semantic pretraining is beneficial not only for ordering visual representations, but also for encoding useful semantic representa- tions of language, both on the word level and the sentence level. 1 Introduction Large-scale "natural language supervision" using image captions collected from the internet has en- abled the first "zero-shot" artificial intelligence (AI) image classifiers, which allow users to create their own image classes using natural language, yet out- perform supervised models on common language- and-image tasks (Radford et al., 2021). The image encoders of such models have been shown to form "multimodal" representations in the upper layers, such that the same neurons fire for photographic, symbolic, and textual depictions of a concept (Goh et al., 2021). Research on these state of the art "vi- sual semantic" (joint language-and-image) models has focused primarily on their benefits for encod- ing semantically legible representations of images. In this paper, we seek to answer a straightforward but as yet unexplored question: what benefits does contrastive visual semantic pretraining have for rep- resentations of natural language? The CLIP ("Contrastive Language Image Pre- training") image classification model introduced by Radford et al. (2021) provides a unique opportunity to observe the effects of visual semantic pretrain- ing on a contextualizing language model. While most other visual semantic architectures combine language and image features in the inner layers of the model (Lu et al., 2019), CLIP separates the lan- …

Content truncated to 500 words