Composition-contrastive Learning for Sentence Embeddings

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 15836–15848 July 9-14, 2023 ©2023 Association for Computational Linguistics Composition-contrastive Learning for Sentence Embeddings Sachin Chanchani and Ruihong Huang Texas A&M University {chanchanis, huangrh}@tamu.edu Abstract Vector representations of natural language are ubiquitous in search applications. Recently, various methods based on contrastive learning have been proposed to learn textual represen- tations from unlabelled data; by maximizing alignment between minimally-perturbed em- beddings of the same text, and encouraging a uniform distribution of embeddings across a broader corpus. Differently, we propose maxi- mizing alignment between texts and a composi- tion of their phrasal constituents. We consider several realizations of this objective and elabo- rate the impact on representations in each case. Experimental results on semantic textual simi- larity tasks show improvements over baselines that are comparable with state-of-the-art ap- proaches. Moreover, this work is the first to do so without incurring costs in auxiliary training objectives or additional network parameters.1 1 Introduction Significant progress has been made on the task of learning universal sentence representations that can be used for a variety of natural language processing tasks without task-specific fine-tuning (Conneau et al., 2017, Cer et al., 2018, Kiros et al., 2015, Lo- geswaran and Lee, 2018, Giorgi et al., 2021a, Yan et al., 2021, Gao et al., 2021, Chuang et al., 2022a). Recent works have shown the potential to learn good sentence embeddings without labeled data by fine-tuning pre-trained language models (PLMs) using the unsupervised framework introduced in SimCLR (Chen et al., 2020), adapted to the natural language processing (NLP) domain. In computer vision (CV), SimCLR exploits a series of transfor- mations (blurs, crops, color distortions, etc.) to construct positive pairs from otherwise unique data points. A cross entropy objective (InfoNCE; Oord et al., 2018) is then applied to minimize distance 1Code, pre-trained models, and datasets will be available at github.com/perceptiveshawty/CompCSE. between representations originating from the same datum, while maximizing the distance to all other points in a mini-batch. The success of the frame- work in computer vision is due largely to the di- versity of augmentations used for creating positive pairs, which leave the identity of the original exam- ple intact while reducing pairwise mutual informa- tion in the input space (Tian et al., 2020; Wu et al., 2020; Purushwalkam and Gupta, 2020). Constructing positive pairs via discrete augmen- tations have not been effective when applying the same objective to sentence embeddings. In fact, Gao et al. (2021) perform an ablation study of tex- tual augmentations (e.g., cropping, synonym re- placement) and find that training on these pairs hurts downstream performance on semantic textual similarity (STS) tasks. Instead, they observe that minimal (10%) dropout noise can be used to cre- ate positive pairs on-the-fly, and empirically results in stronger representations. This framework rely- ing on nearly identical pairs is known as SimCSE. Since the dropout noise exists as a regularization component of the BERT architecture (Devlin et al., 2019a), explicit augmentations are unnecessary, making …

Content truncated to 500 words