Differentiable Data Augmentation for Contrastive Sentence Representation Learning

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7640 - 7653 December 7-11, 2022 ©2022 Association for Computational Linguistics Differentiable Data Augmentation for Contrastive Sentence Representation Learning Tianduo Wang and Wei Lu StatNLP Research Group Singapore University of Technology and Design {tianduo_wang,luwei}@sutd.edu.sg Abstract Fine-tuning a pre-trained language model via the contrastive learning framework with a large amount of unlabeled sentences or labeled sen- tence pairs is a common way to obtain high- quality sentence representations. Although the contrastive learning framework has shown its superiority on sentence representation learning over previous methods, the potential of such a framework is under-explored so far due to the simple method it used to construct posi- tive pairs. Motivated by this, we propose a method that makes hard positives from the orig- inal training examples. A pivotal ingredient of our approach is the use of prefix that is at- tached to a pre-trained language model, which allows for differentiable data augmentation dur- ing contrastive learning. Our method can be summarized in two steps: supervised prefix- tuning followed by joint contrastive fine-tuning with unlabeled or labeled examples. Our ex- periments confirm the effectiveness of our data augmentation approach. The proposed method yields significant improvements over existing methods under both semi-supervised and super- vised settings. Our experiments under a low labeled data setting also show that our method is more label-efficient than the state-of-the-art contrastive learning methods. 1 1 Introduction Learning universal and effective sentence represen- tations is an enduring problem in natural language processing (NLP). The objective of learning sen- tence representations is similar to that of learning word embeddings (Mikolov et al., 2013; Penning- ton et al., 2014), i.e., we expect the embeddings of sentences with similar semantics to be close to each other, while the embeddings of sentences with different meanings to be sufficiently far away from each other. Recently, we have witnessed the suc- cess of pre-trained language models in various NLP 1Our code and model checkpoints are available at https: //github.com/TianduoWang/DiffAug. Stage 1: Prefix-tuning Stage 2: Joint tuning Language model Prefix Pre-training Module-compatible! Figure 1: Our two-stage tuning strategy. tasks. However, the quality of the sentence repre- sentations directly obtained from non fine-tuned language models remains unsatisfactory (Reimers and Gurevych, 2019; Li et al., 2020). One approach to addressing this problem is to fine-tune a pre-trained language model via the con- trastive learning objective on labeled or unlabeled sentences. Several recent research efforts (Yan et al., 2021; Gao et al., 2021b; Jiang et al., 2022) have shown that a contrastive learning based fine- tuning stage can produce state-of-the-art results on the sentence embedding learning task. Some previous contrastive learning works on other modalities, e.g., image (Chen et al., 2020a,b), have shown that an effective data augmentation (DA) method is crucial for the success of con- trastive learning. However, due to the discrete nature of language, applying commonly-used sen- tence augmentation strategies, e.g., word deletion and replacement, over the input sentences for con- trastive learning leads to suboptimal …

Content truncated to 500 words