Alleviating Over-smoothing for Unsupervised Sentence Representation

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 3552–3566 July 9-14, 2023 ©2023 Association for Computational Linguistics Alleviating Over-smoothing for Unsupervised Sentence Representation Nuo Chen1, Linjun Shou2, Jian Pei3, Ming Gong2, Bowen Cao4 Jianhui Chang4, Daxin Jiang2, Jia Li1∗ 1Hong Kong University of Science and Technology (Guangzhou), Hong Kong University of Science and Technology 2STCA, Microsoft, Beijing, 3Duke University, USA 4Peking University, China chennuo26@gmail.com, jialee@ust.hk Abstract Currently, learning better unsupervised sen- tence representations is the pursuit of many natural language processing communities. Lots of approaches based on pre-trained language models (PLMs) and contrastive learning have achieved promising results on this task. Experi- mentally, we observe that the over-smoothing problem reduces the capacity of these pow- erful PLMs, leading to sub-optimal sentence representations. In this paper, we present a Simple method named Self-Contrastive Learning (SSCL) to alleviate this issue, which samples negatives from PLMs intermediate lay- ers, improving the quality of the sentence rep- resentation. Our proposed method is quite simple and can be easily extended to vari- ous state-of-the-art models for performance boosting, which can be seen as a plug-and- play contrastive framework for learning un- supervised sentence representation. Exten- sive results prove that SSCL brings the supe- rior performance improvements of different strong baselines (e.g., BERT and SimCSE) on Semantic Textual Similarity and Transfer datasets. Our codes are available at https: //github.com/nuochenpku/SSCL. 1 Introduction Learning effective sentence representations is a long-standing and fundamental goal of natural lan- guage processing (NLP) communities (Hill et al., 2016; Conneau et al., 2017; Kim et al., 2021; Gao et al., 2021; You et al., 2022), which can be applied to various downstream NLP tasks such as Seman- tic Textual Similarity (Agirre et al., 2012, 2013, 2014, 2015, 2016; Cer et al., 2017; Marelli et al., 2014) and information retrieval (Xiong et al., 2021; Li et al., 2022). Compared with supervised sen- tence representations, unsupervised sentence repre- sentation learning is more challenging because of lacking enough supervised signals. ∗Corresponding Author 0 2 4 6 8 10 Inter-layer Index 0.6 0.7 0.8 0.9 1.0 Inter-layer Similarity SimCSE Ours Figure 1: Inter-layer cosine similarity of sentence rep- resentations computed from SimCSE and Ours. We calculate the sentence representation similarity between two adjacent layers on STS-B test set. In this exam- ple, we extend our methods of SimCSE by utilizing the penultimate layer as negatives. In the context of unsupervised sentence repre- sentation learning, prior works (Devlin et al., 2018; Lan et al., 2020) tend to directly utilize large pre- trained language models (PLMs) as the sentence encoder to achieve promising results. Recently, re- searchers point that the representations from these PLMs suffer from the anisotropy (Li et al., 2020; Su et al., 2021) issue, which denotes the learned representations are always distributed into a narrow one in the semantic space. More recently, several works (Giorgi et al., 2021; Gao et al., 2021) prove that incorporating PLMs with contrastive learning can alleviate this problem, leading to the distribu- …

Content truncated to 500 words