A Contrastive Framework for Learning Sentence Representations from Pairwise and Triple-wise Perspective in Angular Space

A Contrastive Framework for Learning Sentence Representations from Pairwise and Triple-wise Perspective in Angular Space Yuhao Zhang1, Hongji Zhu1, Yongliang Wang1, Nan Xu2, Xiaobo Li1, BinQiang Zhao1 1Alibaba Group 2Institute of Automation, Chinese Academy of Sciences {zyh277500,zhj283587,wangyongliang.wyl,xiaobo.lixb,binqiang.zhao}@alibaba-inc.com xunan2015@ia.ac.cn Abstract Learning high-quality sentence representations is a fundamental problem of natural language processing which could benefit a wide range of downstream tasks. Though the BERT-like pre- trained language models have achieved great success, using their sentence representations directly often results in poor performance on the semantic textual similarity task. Recently, several contrastive learning methods have been proposed for learning sentence representations and have shown promising results. However, most of them focus on the constitution of posi- tive and negative representation pairs and pay little attention to the training objective like NT-Xent, which is not sufficient enough to ac- quire the discriminating power and is unable to model the partial order of semantics between sentences. So in this paper, we propose a new method ArcCSE, with training objectives de- signed to enhance the pairwise discriminative power and model the entailment relation of triplet sentences. We conduct extensive exper- iments which demonstrate that our approach outperforms the previous state-of-the-art on di- verse sentence related tasks, including STS and SentEval. 1 Introduction Learning sentence representations, which encodes sentences into fixed-sized dense vectors such that semantically similar ones stay close, is a fundamen- tal problem of natural language processing. It could benefit a wide range of downstream applications such as information retrieval, semantic similarity comparison, question answering, and so on. Recently, with the great success of pre-trained Transformer-based language models (Devlin et al., 2019; Liu et al., 2019; Raffel et al., 2020; Brown et al., 2020; Liu et al., 2019) like BERT, they have been widely adopted for generating sentence rep- resentations. A straightforward way is by lever- aging the [CLS] embedding (Devlin et al., 2019) −1 0 1 (a) BERTbase −1 0 1 −1 0 1 (b) SimCSE-BERTbase −1 0 1 sa sb sc −1 0 1 (c) ArcCSE-BERTbase −1 0 1 (d) Example Sentences sa: He was born in Nazareth-Palestine, but immigrated to Lebanon with his parents and then to Jordan where he completed his primary education. sb: He was born in Nazareth-Palestine, but immigrated to Lebanon with his parents. sc: He was born in Nazareth-Palestine, but immigrated to Lebanon. Figure 1: Sentence representations visualization. We generate the representations of three related sentences by passing them to BERTbase, SimCSE-BERTbase and ArcCSE-BERTbase multiple times. With different dropout masks, we can generate different representa- tions for each sentence. Then we normalize the embed- dings and use t-SNE for dimensionality reduction. or applying mean pooling on the last layers of a BERT-like pre-trained language model (Reimers and Gurevych, 2019). However, the sentence embeddings coming from a pre-trained language model without further fine-tuning could not cap- ture the semantic meaning of sentences very well as shown in Figure 1(a), and sometimes even un- derperform non-contextualized embeddings like GloVe (Pennington et al., 2014). To make pre-trained language models more …

Content truncated to 500 words