Debiased Contrastive Learning of Unsupervised Sentence Representations

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 6120 - 6130 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Debiased Contrastive Learning of Unsupervised Sentence Representations Kun Zhou1,3, Beichen Zhang2, Wayne Xin Zhao2,3† and Ji-Rong Wen2,3 1School of Information, Renmin University of China. 2Gaoling School of Artificial Intelligence, Renmin University of China 3Beijing Key Laboratory of Big Data Management and Analysis Methods francis_kun_zhou@163.com, {zhangbeichen724,batmanfly}@gmail.com, jrwen@ruc.edu.cn Abstract Recently, contrastive learning has been shown to be effective in improving pre-trained lan- guage models (PLM) to derive high-quality sentence representations. It aims to pull close positive examples to enhance the alignment while push apart irrelevant negatives for the uniformity of the whole representation space. However, previous works mostly adopt in-batch negatives or sample from training data at ran- dom. Such a way may cause the sampling bias that improper negatives (e.g., false neg- atives and anisotropy representations) are used to learn sentence representations, which will hurt the uniformity of the representation space. To address it, we present a new framework DCLR (Debiased Contrastive Learning of un- supervised sentence Representations) to allevi- ate the influence of these improper negatives. In DCLR, we design an instance weighting method to punish false negatives and gener- ate noise-based negatives to guarantee the uni- formity of the representation space. Exper- iments on seven semantic textual similarity tasks show that our approach is more effec- tive than competitive baselines. Our code and data are publicly available at the link: https: //github.com/RUCAIBox/DCLR. 1 Introduction As a fundamental task in the natural language pro- cessing (NLP) field, unsupervised sentence repre- sentation learning (Kiros et al., 2015; Hill et al., 2016) aims to derive high-quality sentence rep- resentations that can benefit various downstream tasks, especially for low-resourced domains or com- putationally expensive tasks, e.g., zero-shot text se- mantic matching (Qiao et al., 2016), large-scale se- mantic similarity comparison (Agirre et al., 2015), and document retrieval (Le and Mikolov, 2014). Recently, pre-trained language models (PLMs) (Devlin et al., 2019) have become a widely-used se- † Corresponding author 0-0.1 0.1-0.2 0.2-0.3 0.3-0.4 0.4-0.5 0.5-0.6 0.6-0.7 0.7-0.8 0.8-0.9 0.9-1.0 0 10 20 30 40 50 Input: They are mostly migrants and are involved in fishing and cultivation for their livelihood Figure 1: The distribution of cosine similarity between an input sentence and 255 in-batch negatives from the commonly-used Wikipedia Corpus. It is evaluated by the SimCSE model (Gao et al., 2021). Almost half of the negatives have high similarities with the input. mantic representation approach, achieving remark- able performance on various NLP tasks. However, several studies have found that the native sentence representations derived by PLMs are not uniformly distributed with respect to directions, but instead oc- cupy a narrow cone in the vector space (Ethayarajh, 2019), which largely limits their expressiveness. To address this issue, contrastive learning (Chen et al., 2020) has been adopted to refine PLM-derived sen- tence representations. It pulls semantically-close neighbors together to improve the alignment, while pushes apart non-neighbors for the …

Content truncated to 500 words