DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4207 - 4218 July 10-15, 2022 ©2022 Association for Computational Linguistics DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings Yung-Sung Chuang† Rumen Dangovski† Hongyin Luo† Yang Zhang‡ Shiyu Chang∗ Marin Soljaˇci´c† Shang-Wen Li⋄ Wen-tau Yih⋄ Yoon Kim† James Glass† Massachusetts Institute of Technology† Meta AI⋄ MIT-IBM Watson AI Lab‡ UC Santa Barbara∗ yungsung@mit.edu Abstract We propose DiffCSE, an unsupervised con- trastive learning framework for learning sen- tence embeddings. DiffCSE learns sentence embeddings that are sensitive to the difference between the original sentence and an edited sen- tence, where the edited sentence is obtained by stochastically masking out the original sentence and then sampling from a masked language model. We show that DiffSCE is an instance of equivariant contrastive learning (Dangovski et al., 2021), which generalizes contrastive learning and learns representations that are in- sensitive to certain types of augmentations and sensitive to other “harmful” types of augmen- tations. Our experiments show that DiffCSE achieves state-of-the-art results among unsuper- vised sentence representation learning methods, outperforming unsupervised SimCSE1 by 2.3 absolute points on semantic textual similarity tasks. 2 1 Introduction Learning “universal” sentence representations that capture rich semantic information and are at the same time performant across a wide range of down- stream NLP tasks without task-specific finetuning is an important open issue in the field (Conneau et al., 2017; Cer et al., 2018; Kiros et al., 2015; Logeswaran and Lee, 2018; Giorgi et al., 2020; Yan et al., 2021; Gao et al., 2021). Recent work has shown that finetuning pretrained language mod- els with contrastive learning makes it possible to learn good sentence embeddings without any la- beled data (Giorgi et al., 2020; Yan et al., 2021; Gao et al., 2021). Contrastive learning uses multi- ple augmentations on a single datum to construct positive pairs whose representations are trained to 1SimCSE has two settings: unsupervised and supervised. In this paper, we focus on the unsupervised setting. Unless otherwise stated, in this paper we use SimCSE to refer to unsupervised SimCSE. 2Pretrained models and code are available at https:// github.com/voidism/DiffCSE. be more similar to one another than negative pairs. While different data augmentations (random crop- ping, color jitter, rotations, etc.) have been found to be crucial for pretraining vision models (Chen et al., 2020), such augmentations have generally been un- successful when applied to contrastive learning of sentence embeddings. Indeed, Gao et al. (2021) find that constructing positive pairs via a simple dropout-based augmentation works much better than more complex augmentations such as word deletions or replacements based on synonyms or masked language models. This is perhaps unsur- prising in hindsight; while the training objective in contrastive learning encourages representations to be invariant to augmentation transformations, direct augmentations on the input (e.g., deletion, replacement) often change the meaning of the sen- tence. That is, ideal sentence embeddings should not be invariant to such transformations. We propose to learn sentence representations that are aware …

Content truncated to 500 words