Pairwise Supervised Contrastive Learning of Sentence Representations

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5786–5798 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 5786 Pairwise Supervised Contrastive Learning of Sentence Representations Dejiao Zhang Shang-Wen Li∗ Wei Xiao∗ Henghui Zhu∗ Ramesh Nallapati Andrew O. Arnold Bing Xiang AWS AI Abstract Many recent successes in sentence represen- tation learning have been achieved by sim- ply fine-tuning on the Natural Language In- ference (NLI) datasets with triplet loss or siamese loss. Nevertheless, they share a com- mon weakness: sentences in a contradiction pair are not necessarily from different seman- tic categories. Therefore, optimizing the se- mantic entailment and contradiction reasoning objective alone is inadequate to capture the high-level semantic structure. The drawback is compounded by the fact that the vanilla siamese or triplet losses only learn from in- dividual sentence pairs or triplets, which of- ten suffer from bad local optima. In this pa- per, we propose PairSupCon, an instance dis- crimination based approach aiming to bridge semantic entailment and contradiction under- standing with high-level categorical concept encoding. We evaluate PairSupCon on vari- ous downstream tasks that involve understand- ing sentence semantics at different granulari- ties. We outperform the previous state-of-the- art method with 10%–13% averaged improve- ment on eight clustering tasks, and 5%–6% av- eraged improvement on seven semantic textual similarity (STS) tasks. 1 Introduction Learning high-quality sentence embeddings is a fundamental task in Natural Language Processing. The goal is to map semantically similar sentences close together and dissimilar sentences farther apart in the representation space. Many recent successes have been achieved by training on the NLI datasets (Bowman et al., 2015; Williams et al., 2017; Wang et al., 2018), where the task is often to classify each sentence pair into one of three categories: entail- ment, contradiction, or neutral. ∗Equal contribution. Correspondence to Dejiao Zhang <dejiaoz@amazon.com>. We will release the code at https: //github.com/amazon-research/ STS12 STS13 STS14 STS15 STS16 SICK-R STS-B 0.65 0.75 0.85 SBERT PairSupCon (a) Spearman Correlation Evaluation on STS. SBERT PairSupCon (b) TSNE visualization of the StackOverflow embeddings. Figure 1: Simultaneoulsy encoding semantic categori- cal structure and pairwise entailment and contradiction understanding into embeddings. Despite promising results, prior work (Con- neau et al., 2017; Cer et al., 2018; Reimers and Gurevych, 2019a) share a common weakness: the sentences forming a contradiction pair may not necessarily belong to different semantic categories. Consequently, optimizing the model for semantic entailment and contradiction understanding alone is inadequate to encode the high-level categori- cal concepts into the representations. Moreover, the vanilla siamese (triplet) loss only learns from the individual sentence pairs (triplets), which often requires substantial training examples to achieve competitive performance (Oh Song et al., 2016; Thakur et al., 2020). As shown in Section 4.1, the siamese loss can sometimes drive a model to bad local optima where the performance of high-level semantic concept encoding is degraded when com- pared with its baseline counterpart. In this paper, we take inspiration from self- supervised contrastive learning (Bachman et al., 5787 2019; He et al., …

Content truncated to 500 words