RankCSE: Unsupervised Sentence Representations Learning via Learning to Rank

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 13785–13802 July 9-14, 2023 ©2023 Association for Computational Linguistics RankCSE : Unsupervised Sentence Representation Learning via Learning to Rank Jiduan Liu1,2∗, Jiahao Liu3, Qifan Wang4, Jingang Wang3, Wei Wu3 Yunsen Xian3, Dongyan Zhao1,2,5,6†, Kai Chen7, Rui Yan8,9† 1Wangxuan Institute of Computer Technology, Peking University 2Center for Data Science, AAIS, Peking University; 3Meituan; 4Meta AI 5National Key Laboratory of General Artificial Intelligence 6BIGAI, Beijing, China; 7School of Economics, Peking University 8Gaoling School of Artificial Intelligence, Renmin University of China 9Engineering Research Center of Next-Generation Intelligent Search and Recommendation, Ministry of Education {liujiduan,chen.kai,zhaody}@pku.edu.cn, ruiyan@ruc.edu.cn, wqfcr@fb.com {liujiahao12,wangjingang02,xianyunsen}@meituan.com, wuwei19850318@gmail.com Abstract Unsupervised sentence representation learning is one of the fundamental problems in natural language processing with various downstream applications. Recently, contrastive learning has been widely adopted which derives high- quality sentence representations by pulling sim- ilar semantics closer and pushing dissimilar ones away. However, these methods fail to capture the fine-grained ranking information among the sentences, where each sentence is only treated as either positive or negative. In many real-world scenarios, one needs to distin- guish and rank the sentences based on their sim- ilarities to a query sentence, e.g., very relevant, moderate relevant, less relevant, irrelevant, etc. In this paper, we propose a novel approach, RankCSE, for unsupervised sentence repre- sentation learning, which incorporates ranking consistency and ranking distillation with con- trastive learning into a unified framework. In particular, we learn semantically discriminative sentence representations by simultaneously en- suring ranking consistency between two repre- sentations with different dropout masks, and distilling listwise ranking knowledge from the teacher. An extensive set of experiments are conducted on both semantic textual similarity (STS) and transfer (TR) tasks. Experimental results demonstrate the superior performance of our approach over several state-of-the-art baselines. 1 Introduction Sentence representation learning refers to the task of encoding sentences into fixed-dimensional em- †∗ Work done during internship at Meituan. Corresponding authors: Dongyan Zhao (zhaody@pku.edu.cn) and Rui Yan (ruiyan@ruc.edu.cn). Target Sentences Label SimCSE RankCSE • A woman is breaking eggs 4.80 (1) 0.93 (2) 0.97 (1) • A man is cracking eggs 3.60 (2) 0.94 (1) 0.91 (2) • A woman is talking to a man 1.60 (3) 0.45 (5) 0.65 (3) • A man and a woman are speaking 1.40 (4) 0.47 (3) 0.61 (4) • A man is talking to a boy 1.00 (5) 0.46 (4) 0.56 (5) Query Sentence: A woman is cracking eggs • Broccoli are being cut by a woman 4.80 (1) 0.82 (2) 0.95 (1) • A woman is slicing vegetables 4.20 (2) 0.83 (1) 0.91 (2) • A woman is cutting some plants 3.50 (3) 0.74 (5) 0.87 (3) • There is no woman cutting broccoli 3.40 (4) 0.76 (3) 0.85 (4) • A woman is cutting some flowers 2.87 (5) 0.71 (7) 0.81 (5) • A man is slicing tomatoes 2.60 (6) 0.75 (4) 0.79 (6) • A man is cutting tomatoes 2.40 (7) 0.73 (6) 0.76 (7) Query Sentence: …

Content truncated to 500 words