Exploring the Impact of Negative Samples of Contrastive Learning: A Case Study of Sentence Embedding

Findings of the Association for Computational Linguistics: ACL 2022, pages 3138 - 3152 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Exploring the Impact of Negative Samples of Contrastive Learning: A Case Study of Sentence Embedding Rui Cao1, * and Yihao Wang1, * and Yuxin Liang1 Ling Gao1, † and Jie Zheng1 and Jie Ren 2 and Zheng Wang 3 1Northwest University, 2Shaanxi Normal University, and 3University of Leeds {caorui, wangyihao, liangyuxin}@stumail.nwu.edu.cn, {gl, jzheng}@nwu.edu.cn, renjie@snnu.edu.cn, z.wang5@leeds.ac.uk Abstract Contrastive learning is emerging as a powerful technique for extracting knowledge from unla- beled data. This technique requires a balanced mixture of two ingredients: positive (similar) and negative (dissimilar) samples. This is typ- ically achieved by maintaining a queue of neg- ative samples during training. Prior works in the area typically uses a fixed-length negative sample queue, but how the negative sample size affects the model performance remains un- clear. The opaque impact of the number of neg- ative samples on performance when employ- ing contrastive learning aroused our in-depth exploration. This paper presents a momen- tum contrastive learning model with negative sample queue for sentence embedding, namely MoCoSE. We add the prediction layer to the online branch to make the model asymmetric and together with EMA update mechanism of the target branch to prevent the model from collapsing. We define a maximum traceable distance metric, through which we learn to what extent the text contrastive learning bene- fits from the historical information of negative samples. Our experiments find that the best results are obtained when the maximum trace- able distance is at a certain range, demonstrat- ing that there is an optimal range of historical information for a negative sample queue. We evaluate the proposed unsupervised MoCoSE on the semantic text similarity (STS) task and obtain an average Spearman’s correlation of 77.27%. Source code is available here. 1 Introduction In recent years, unsupervised learning has been brought to the fore in deep learning due to its ability to leverage large-scale unlabeled data. Various un- supervised contrastive models is emerging, continu- ously narrowing down the gap between supervised and unsupervised learning. Contrastive learning *Authors contributed equally to this manuscript. †Corresponding author. suffers from the problem of model collapse, where the model converges to a constant value and the samples all mapped to a single point in the feature space. Negative samples are an effective way to solve this problem. In computer vision, SimCLR from Chen (Chen et al., 2020) and MoCo from He (He et al., 2020) is known for using negative samples and get the leading performance in the contrastive learning. SimCLR uses different data augmentation (e.g., rotation, masking, etc.) on the same image to con- struct positive samples, and negative samples are from the rest of images in the same batch. MoCo goes a step further by randomly select the data in entire unlabeled training set to stack up a first-in- first-out negative sample queue. Recently in natural language processing, con- trastive learning has been widely used in the task of …

Content truncated to 500 words