ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 5065–5075 August 1–6, 2021. ©2021 Association for Computational Linguistics 5065 ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer Yuanmeng Yan1∗, Rumei Li2∗, Sirui Wang2, Fuzheng Zhang2, Wei Wu2, Weiran Xu1 1Beijing University of Posts and Telecommunications, Beijing, China 2Meituan Inc., Beijing, China {yanyuanmeng,xuweiran}@bupt.edu.cn {lirumei,wangsirui,zhangfuzheng,wuwei30}@meituan.com Abstract Learning high-quality sentence representa- tions benefits a wide range of natural language processing tasks. Though BERT-based pre- trained language models achieve high perfor- mance on many downstream tasks, the native derived sentence representations are proved to be collapsed and thus produce a poor perfor- mance on the semantic textual similarity (STS) tasks. In this paper, we present ConSERT, a Contrastive Framework for Self-Supervised SEntence Representation Transfer, that adopts contrastive learning to fine-tune BERT in an unsupervised and effective way. By making use of unlabeled texts, ConSERT solves the collapse issue of BERT-derived sentence rep- resentations and make them more applicable for downstream tasks. Experiments on STS datasets demonstrate that ConSERT achieves an 8% relative improvement over the previous state-of-the-art, even comparable to the super- vised SBERT-NLI. And when further incorpo- rating NLI supervision, we achieve new state- of-the-art performance on STS tasks. More- over, ConSERT obtains comparable results with only 1000 samples available, showing its robustness in data scarcity scenarios. 1 Introduction Sentence representation learning plays a vital role in natural language processing tasks (Kiros et al., 2015; Hill et al., 2016; Conneau et al., 2017; Cer et al., 2018). Good sentence representations benefit a wide range of downstream tasks, especially for computationally expensive ones, including large- scale semantic similarity comparison and informa- tion retrieval. Recently, BERT-based pre-trained language models have achieved high performance on many ∗Work done during internship at Meituan Inc. The first two authors contribute equally. Weiran Xu is the correspond- ing author. 0 1 2 3 4 5 the gold similarity score 0.2 0.0 0.2 0.4 0.6 0.8 1.0 the predicted cosine similarity a) BERT derived sentence representation 0 1 2 3 4 5 the gold similarity score 0.2 0.0 0.2 0.4 0.6 0.8 1.0 the predicted cosine similarity b) After applying our approach Figure 1: The correlation diagram between the gold similarity score (x-axis) and the model predicted cosine similarity score (y-axis) on the STS benchmark dataset. downstream tasks with additional supervision. However, the native sentence representations de- rived from BERT1 are proved to be of low-quality (Reimers and Gurevych, 2019; Li et al., 2020). As shown in Figure 1a, when directly adopt BERT- based sentence representations to semantic textual similarity (STS) tasks, almost all pairs of sentences achieved a similarity score between 0.6 to 1.0 , even if some pairs are regarded as completely unre- lated by the human annotators. In other words, the BERT-derived native sentence representations are somehow collapsed (Chen and He, 2020), which means almost all sentences are mapped into a small area and therefore produce high similarity. Such phenomenon is also …

Content truncated to 500 words