PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 12052 - 12066 December 7-11, 2022 ©2022 Association for Computational Linguistics PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings Qiyu Wu1∗, Chongyang Tao2, Tao Shen2, Can Xu2, Xiubo Geng2, Daxin Jiang2† 1The University of Tokyo, Tokyo, Japan 2Microsoft Corporation 1qiyuw@g.ecc.u-tokyo.ac.jp 2{chotao,shentao,caxu,xigeng,djiang}@microsoft.com Abstract Learning sentence embeddings in an unsuper- vised manner is fundamental in natural lan- guage processing. Recent common practice is to couple pre-trained language models with unsupervised contrastive learning, whose suc- cess relies on augmenting a sentence with a semantically-close positive instance to con- struct contrastive pairs. Nonetheless, exist- ing approaches usually depend on a mono- augmenting strategy, which causes learning shortcuts towards the augmenting biases and thus corrupts the quality of sentence embed- dings. A straightforward solution is resort- ing to more diverse positives from a multi- augmenting strategy, while an open question re- mains about how to unsupervisedly learn from the diverse positives but with uneven augment- ing qualities in the text field. As one answer, we propose a novel Peer-Contrastive Learning (PCL) with diverse augmentations. PCL con- structs diverse contrastive positives and nega- tives at the group level for unsupervised sen- tence embeddings. PCL performs peer-positive contrast as well as peer-network cooperation, which offers an inherent anti-bias ability and an effective way to learn from diverse augmenta- tions. Experiments on STS benchmarks verify the effectiveness of PCL against its competitors in unsupervised sentence embeddings.1 1 Introduction Sentence embedding learning, which aims at deriv- ing semantically meaningful fixed-sized vectors for sentences, is a natural language processing (NLP) technique of great significance, especially for time-sensitive downstream tasks (Reimers and Gurevych, 2019). Recently, contrastive learning (CL) is proven effective to learn representation (Wu et al., 2018; Tian et al., 2020; He et al., 2020) and ∗Work done during the internship at Microsoft. † Corresponding author. 1Our implementation is available at https://github. com/qiyuw/PeerCL. Augmenting Order N-gram Bag-of-words Shuffled Sentence × × ✓ Inversed Sentence × ✓ ✓ Word Repetition ✓ × ✓ Word Deletion ✓ × × Table 1: Text augmentation strategies change semantics in the sentence but still has shortcuts to learn. Employing limited number of strategies causes learning shortcuts towards the augmenting bias. substantially improve its performance (Yan et al., 2021; Gao et al., 2021) when coupling with pre- trained language models (PLMs). The main idea of contrastive learning for sentence embedding is pulling semantic neighbors together and pushing semantic non-neighbors apart (Hadsell et al., 2006), which naturally requires effective contrastive pairs. As effective contrastive pairs are usually scarce and require much human effort to collect, how to learn sentence embeddings in an fully unsupervised manner has become a challenging yet attractive re- search area (Wang et al., 2021; Giorgi et al., 2021). The key to unsupervised contrastive learning for sentence embedding is to augment a given anchor sentence with an effective positive instance to con- struct the pairs. Hence, many efforts have been made to design augmentation methods by adding noises or …

Content truncated to 500 words