WhitenedCSE: Whitening-based Contrastive Learning of Sentence Embeddings

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 12135–12148 July 9-14, 2023 ©2023 Association for Computational Linguistics WhitenedCSE: Whitening-based Contrastive Learning of Sentence Embeddings Wenjie Zhuo1 Yifan Sun2 Xiaohan Wang1 Linchao Zhu1 Yi Yang1† 1Zhejiang University, Hangzhou, China 2Baidu Inc., Beijing, China {12021057,yangyics}@zju.edu.cn sunyf15@tsinghua.org.cn {wxh1996111, zhulinchao7}@gmail.com Abstract This paper presents a whitening-based con- trastive learning method for sentence embed- ding learning (WhitenedCSE), which combines contrastive learning with a novel shuffled group whitening. Generally, contrastive learning pulls distortions of a single sample (i.e., positive sam- ples) close and push negative samples far away, correspondingly facilitating the alignment and uniformity in the feature space. A popular alter- native to the “pushing” operation is whitening the feature space, which scatters all the sam- ples for uniformity. Since the whitening and the contrastive learning have large redundancy w.r.t. the uniformity, they are usually used sep- arately and do not easily work together. For the first time, this paper integrates whitening into the contrastive learning scheme and facilitates two benefits. 1) Better uniformity. We find that these two approaches are not totally redun- dant but actually have some complementarity due to different uniformity mechanism. 2) Bet- ter alignment. We randomly divide the feature into multiple groups along the channel axis and perform whitening independently within each group. By shuffling the group division, we de- rive multiple distortions of a single sample and thus increase the positive sample diversity. Con- sequently, using multiple positive samples with enhanced diversity further improves contrastive learning due to better alignment. Extensive ex- periments on seven semantic textual similarity tasks show our method achieves consistent im- provement over the contrastive learning base- line and sets new states of the art, e.g., 78.78% (+2.53% based on BERTbase) Spearman corre- lation on STS tasks.1 1 Introduction This paper considers self-supervised sentence rep- resentation (embedding) learning. It is a funda- mental task in language processing (NLP) and can †Corresponding author. 1Our code will be available at https://github.com/ SupstarZh/WhitenedCSE. (a) bert (b) bert+contrastive (c) bert+SGW+contrastive (d) positive distribution Figure 1: The uniformity gradually improves in the deep embedding of (a) bert, (b) bert + contrastive, and (c) bert + SGW + contrastive, i.e., the proposed WhitenedCSE. Meanwhile, in (d), the positive samples after SGW (red) obtain higher diversity than the original bert features (green). Using these diverse positive samples for con- trastive learning, the proposed WhitenedCSE achieves better alignment. benefit a wide range of downstream tasks (Qiao et al., 2016; Le and Mikolov, 2014; Lan et al., 2019; Logeswaran and Lee, 2018). Two characteristics matter for sentence embeddings, i.e., uniformity (of the overall feature distribution) and alignment (of the positive samples), according to a common sense in deep representation learning (Wang and Isola, 2020). Alignment expects minimal distance be- tween positive pairs, while uniformity expects the features are uniformly distributed in the representa- tion space in overall. From this viewpoint, the pop- ular masked language modeling (MLM) (Devlin et al., 2018; Liu et al., 2019; Brown …

Content truncated to 500 words