SimCSE: Simple Contrastive Learning of Sentence Embeddings

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6894–6910 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 6894 SimCSE: Simple Contrastive Learning of Sentence Embeddings Tianyu Gao†∗ Xingcheng Yao‡∗ Danqi Chen† †Department of Computer Science, Princeton University ‡Institute for Interdisciplinary Information Sciences, Tsinghua University {tianyug,danqic}@cs.princeton.edu yxc18@mails.tsinghua.edu.cn Abstract This paper presents SimCSE, a simple con- trastive learning framework that greatly ad- vances the state-of-the-art sentence embed- dings. We first describe an unsupervised ap- proach, which takes an input sentence and predicts itself in a contrastive objective, with only standard dropout used as noise. This simple method works surprisingly well, per- forming on par with previous supervised coun- terparts. We find that dropout acts as mini- mal data augmentation and removing it leads to a representation collapse. Then, we pro- pose a supervised approach, which incorpo- rates annotated pairs from natural language inference datasets into our contrastive learn- ing framework, by using “entailment” pairs as positives and “contradiction” pairs as hard negatives. We evaluate SimCSE on standard semantic textual similarity (STS) tasks, and our unsupervised and supervised models using BERTbase achieve an average of 76.3% and 81.6% Spearman’s correlation respectively, a 4.2% and 2.2% improvement compared to previous best results. We also show—both theoretically and empirically—that contrastive learning objective regularizes pre-trained em- beddings’ anisotropic space to be more uni- form, and it better aligns positive pairs when supervised signals are available.1 1 Introduction Learning universal sentence embeddings is a fun- damental problem in natural language process- ing and has been studied extensively in the litera- ture (Kiros et al., 2015; Hill et al., 2016; Conneau et al., 2017; Logeswaran and Lee, 2018; Cer et al., 2018; Reimers and Gurevych, 2019, inter alia). In this work, we advance state-of-the-art sentence *The first two authors contributed equally (listed in alpha- betical order). This work was done when Xingcheng visited the Princeton NLP group remotely. 1Our code and pre-trained models are publicly available at https://github.com/princeton-nlp/SimCSE. embedding methods and demonstrate that a con- trastive objective can be extremely effective when coupled with pre-trained language models such as BERT (Devlin et al., 2019) or RoBERTa (Liu et al., 2019). We present SimCSE, a simple contrastive sentence embedding framework, which can pro- duce superior sentence embeddings, from either unlabeled or labeled data. Our unsupervised SimCSE simply predicts the input sentence itself with only dropout (Srivastava et al., 2014) used as noise (Figure 1(a)). In other words, we pass the same sentence to the pre-trained encoder twice: by applying the standard dropout twice, we can obtain two different embeddings as “positive pairs”. Then we take other sentences in the same mini-batch as “negatives”, and the model pre- dicts the positive one among negatives. Although it may appear strikingly simple, this approach outper- forms training objectives such as predicting next sentences (Logeswaran and Lee, 2018) and discrete data augmentation (e.g., word deletion and replace- ment) by a large margin, and even matches previous supervised methods. Through careful analysis, we find that dropout acts as minimal …

Content truncated to 500 words