AugCSE: Contrastive Sentence Embedding with Diverse Augmentations

Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 375–398 November 20–23, 2022. ©2022 Association for Computational Linguistics 375 AugCSE: Contrastive Sentence Embedding with Diverse Augmentations Zilu Tang Boston University zilutang@bu.edu Muhammed Yusuf Kocyigit Boston University kocyigit@bu.edu Derry Wijaya Boston University wijaya@bu.edu Abstract Data augmentation techniques have been proven useful in many applications in NLP fields. Most augmentations are task-specific, and cannot be used as a general-purpose tool. In our work, we present AugCSE, a unified framework to utilize diverse sets of data aug- mentations to achieve a better, general pur- pose, sentence embedding model. Building upon the latest sentence embedding models, our approach uses a simple antagonistic dis- criminator that differentiates the augmentation types. With the finetuning objective borrowed from domain adaptation, we show that diverse augmentations, which often lead to conflicting contrastive signals, can be tamed to produce a better and more robust sentence representation. Our methods1 achieve state-of-the-art results on downstream transfer tasks and perform com- petitively on semantic textual similarity tasks, using only unsupervised data. 1 Introduction Data augmentation in NLP can be useful in many situations, from low resource data setting, domain adaptation (Wei et al., 2021), debiasing (Dinan et al., 2020), to improving generalization, robust- ness (Dhole et al., 2021). In the vision domain, Chen et al. (2020b) shows that a diverse set of augmentation can be used to learn a robust general- purpose representation with contrastive learning. Similar work in sentence embedding space (Gao et al. 2021; Chuang et al. 2022) has shown that a simple single augmentation such as dropouts from transformers (Devlin et al., 2019) can be used for contrastive objective. However, no previous work has thoroughly explored the impacts of a diverse set of augmentations with contrastive learning in the sentence embedding space. It is not straightfor- ward to find the best augmentations that work for 1Our code and data can be found at https://github.com/PootieT/AugCSE contrastive learning in different datasets or tasks (Gao et al., 2021). Single augmentation can in- still invariance in models for a specific aspects of linguistic variability, while naively combining a diverse set of augmentations can lead to contradict- ing gradients, preventing models from generalizing well (Table 6)2. In this work, we present AugCSE (Figure 1), a general approach to select and unify a diverse set of augmentations for the purpose of building a general-purpose sentence embedding. During training, in addition to using contrastive loss, we randomly perturb sentences with different augmentations and use a discriminator loss to unify embeddings from diverse augmentations. In short, our work presents the following key contributions: • We show simple data augmentation methods can be used to improve individual tasks, while degrading performance on other tasks (due to shifted domain distribution). • We present our simple discriminator objective that achieves competitive results on sentence similarity task (STS) and transfer classifica- tion tasks against state-of-the-art methods. • We demonstrate through …

Content truncated to 500 words