Virtual Augmentation Supported Contrastive Learning of Sentence Representations

Findings of the Association for Computational Linguistics: ACL 2022, pages 864 - 876 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Virtual Augmentation Supported Contrastive Learning of Sentence Representations Dejiao Zhang∗ Wei Xiao Henghui Zhu Xiaofei Ma Andrew O. Arnold AWS AI Labs, New York Abstract Despite profound successes, contrastive rep- resentation learning relies on carefully de- signed data augmentations using domain- specific knowledge. This challenge is magni- fied in natural language processing, where no general rules exist for data augmentation due to the discrete nature of natural language. We tackle this challenge by presenting a Virtual augmentation Supported Contrastive Learning of sentence representations (VaSCL). Originat- ing from the interpretation that data augmenta- tion essentially constructs the neighborhoods of each training instance, we in turn utilize the neighborhood to generate effective data augmentations. Leveraging the large training batch size of contrastive learning, we approx- imate the neighborhood of an instance via its K-nearest in-batch neighbors in the represen- tation space. We then define an instance dis- crimination task regarding the neighborhood and generate the virtual augmentation in an ad- versarial training manner. We access the per- formance of VaSCL on a wide range of down- stream tasks and set a new state-of-the-art for unsupervised sentence representation learning. 1 Introduction Universal sentence representation learning has been a long-standing problem in Natural Language Processing (NLP). Leveraging the distributed word representations (Bengio et al., 2003; Mikolov et al., 2013; Collobert et al., 2011; Pennington et al., 2014) as the base features to produce sentence representations is a common strategy in the early stage. However, these approaches are tailored to different target tasks, thereby yielding less generic sentence representations (Yessenalina and Cardie, 2011; Socher et al., 2013; Kalchbrenner et al., 2014; Cho et al., 2014). ∗ The code and pretrained checkpoints can be found at https://github.com/amazon-research/ sentence-representations. Correspondence to De- jiao Zhang <dejiaoz@amazon.com>. This issue has motivated more research efforts on designing generic sentence-level learning objec- tives or tasks. Among them, supervised learning on the Natural Language Inference (NLI) datasets (Bowman et al., 2015a; Williams et al., 2017; Wang et al., 2018) has established benchmark transfer learning performance on various downstream tasks (Conneau et al., 2017; Cer et al., 2018; Reimers and Gurevych, 2019a; Zhang et al., 2021). De- spite promising progress, the high cost of collecting annotations precludes its wide applicability, espe- cially when the target domain has scarce annota- tions but differs significantly from the NLI datasets (Zhang et al., 2020). On the other hand, unsupervised learning of sen- tence representations has seen a resurgence of in- terest with the recent successes in self-supervised contrastive learning. These approaches rely on two main components, data augmentation and an instance-level contrastive loss. The popular con- trastive learning objectives Chen et al. (2020); He et al. (2020) and their variants thereof have empiri- cally shown their effectiveness in NLP. However, the discrete nature of the text makes it challeng- ing to establish universal rules for effective text augmentation generation. Various contrastive learning based approaches have …

Content truncated to 500 words