Smoothed Contrastive Learning for Unsupervised Sentence Embedding

Proceedings of the 29th International Conference on Computational Linguistics, pages 4902–4906 October 12–17, 2022. 4902 Smoothed Contrastive Learning for Unsupervised Sentence Embedding Xing Wu1,2,3, Chaochen Gao1,2∗, Yipeng Su1, Jizhong Han1, Zhongyuan Wang3, Songlin Hu1,2† 1Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China 2School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China 3Kuaishou Technology, Beijing, China {wuxing,gaochaochen,suyipeng,hanjizhong,husonglin}@iie.ac.cn wangzhongyuan@kuaishou.com Abstract Unsupervised contrastive sentence embedding models, e.g., unsupervised SimCSE, use the InfoNCE loss function in training. Theoreti- cally, we expect to use larger batches to get more adequate comparisons among samples and avoid overfitting. However, increasing batch size leads to performance degradation when it exceeds a threshold, which is proba- bly due to the introduction of false-negative pairs through statistical observation. To al- leviate this problem, we introduce a simple smoothing strategy upon the InfoNCE loss function, termed Gaussian Smoothed InfoNCE (GS-InfoNCE). In other words, we add ran- dom Gaussian noise as an extension to the negative pairs without increasing the batch size. Through experiments on the semantic text similarity tasks, though simple, the proposed smoothing strategy brings improvements to un- supervised SimCSE. Our code are available at https://github.com/caskcsg/gsInfoNCE. 1 Introduction Good sentence representation benefits many natural language processing tasks, and sentence representa- tion learning has been widely studied (Logeswaran and Lee, 2018; Reimers and Gurevych, 2019). Con- trastive learning has recently been proposed and extensively explored to learn high-quality sentence representations based on the pre-trained language models(Devlin et al., 2018; Liu et al., 2019). Con- trastive learning aims to learn effective representa- tion by pulling close semantically similar sentences while pushing apart dissimilar ones (Hadsell et al., 2006). Among those unsupervised sentence em- bedding learning methods with contrastive learning, the latest state-of-the-art method, as far as we know, is unsupervised SimCSE (unsup-SimCSE) (Gao et al., 2021). unsup-SimCSE implicitly hypothe- sizes “dropout” as minimal data augmentation and ∗The first two authors contribute equally. †Corresponding author. Figure 1: The changing trend of the cosine similarities of the negative pairs in the batch. As the batch_size increases, the mean values of the top 4 cosine similari- ties also increase, indicating negative pairs with lower confidence exists. assumes a sentence is semantically more similar to its augmented counterpart than any other sentence. Though simple, unsup-SimCSE works surprisingly well, performing on par with previously supervised counterparts. Theoretically, since contrastive learning is car- ried out among samples within a batch, increasing the batch size will probably bring more adequate comparisons and avoid overfitting. However, ac- cording to the original unsup-SimCSE paper (Gao et al., 2021), a larger batch size does not always lead to improvements. The performance even de- creases when the batch size exceeds a threshold. We assume that as the batch size increases, more similar sentence samples are probably introduced and easily constitute false-negative pairs, which is detrimental to the learning of the model. We de- sign a probing statistical experiment for different batch sizes to verify our assumption. We use the currently best semantic textual similarity model, i.e., the SimCSE-RoBERTalarge …

Content truncated to 500 words