ESimCSE: Enhanced Sample Building Method for Contrastive Learning of Unsupervised Sentence Embedding

Proceedings of the 29th International Conference on Computational Linguistics, pages 3898–3907 October 12–17, 2022. 3898 ESimCSE: Enhanced Sample Building Method for Contrastive Learning of Unsupervised Sentence Embedding Xing Wu1,2,3, Chaochen Gao1,2∗, Liangjun Zang1, Jizhong Han1, Zhongyuan Wang3, Songlin Hu1,2† 1Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China 2School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China 3Kuaishou Technology, Beijing, China {wuxing,gaochaochen,zangliangjun,hanjizhong,husonglin}@iie.ac.cn wangzhongyuan@kuaishou.com Abstract SimCSE 1 adopts dropout as data augmentation and encodes an input sentence twice into two corresponding embeddings to build a positive pair. Since SimCSE is a Transformer-based encoder that directly encodes the length infor- mation of sentences through positional embed- dings, the two embeddings in a positive pair contain the same length information. Thus, a model trained with these positive pairs is bi- ased, tending to consider that sentences of the same or similar length are more similar in se- mantics. To alleviate it, we apply a simple but effective repetition operation to modify the input sentence. Then we pass the input sen- tence and its modified counterpart to the pre- trained Transformer encoder, respectively, to get the positive pair. Additionally, we draw inspiration from the computer vision commu- nity and introduce momentum contrast to en- large the number of negative pairs without ad- ditional calculations. The proposed modifica- tions are applied to positive and negative pairs separately, and build a new sentence embed- ding method, termed Enhanced SimCSE (ES- imCSE). We evaluate the proposed ESimCSE on several benchmark datasets w.r.t the seman- tic text similarity (STS) task. Experimental results show that ESimCSE outperforms Sim- CSE by an average Spearman correlation of 2.02% on BERT-base. Our code are available at https://github.com/caskcsg/ESimCSE. 1 Introduction Recently, researchers have proposed using con- trastive learning to learn better unsupervised sen- tence embeddings (Wu et al.; Zhang et al., b; Liu et al., 2021; Gao et al., 2021; Yan et al., 2021). Contrastive learning aims to learn effective sen- tence embeddings based on the assumption that ∗The first two authors contribute equally. †Corresponding author. 1We focus on unsupervised sentence embedding, so Sim- CSE in this article refers to unsupervised SimCSE. Length Diff Avg. Similarity Diff > 3 16.34 ≤ 3 18.18 (+1.84) Table 1: The average similarity difference between the model (SimCSE-BERT) predictions and the normalized ground truths. effective sentence embeddings should bring simi- lar sentences closer while pushing away dissimilar ones. It generally uses various data augmentation methods (Shleifer, 2019; Wei and Zou, 2019; Wu et al., 2019) to generate different views for each sen- tence randomly, and assumes a sentence is seman- tically more similar to its augmented counterpart than any other sentence. Among these methods, the most representative one is SimCSE (Gao et al., 2021), which performs on par with previously su- pervised counterparts. SimCSE implicitly hypothe- sizes dropout acts as a minimal data augmentation method. Specifically, SimCSE composes N sen- tences in a batch and feeds each sentence to the pre-trained BERT twice with two independently sampled dropout masks. Then the embeddings de- …

Content truncated to 500 words