miCSE: Mutual Information Contrastive Learning for Low-shot Sentence Embeddings

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 6159–6177 July 9-14, 2023 ©2023 Association for Computational Linguistics miCSE: Mutual Information Contrastive Learning for Low-shot Sentence Embeddings Tassilo Klein SAP AI Research tassilo.klein@sap.com Moin Nabi SAP AI Research m.nabi@sap.com Abstract This paper presents miCSE, a mutual information-based contrastive learning framework that significantly advances the state- of-the-art in few-shot sentence embedding. The proposed approach imposes alignment between the attention pattern of different views during contrastive learning. Learning sentence embeddings with miCSE entails enforcing the structural consistency across augmented views for every sentence, making contrastive self-supervised learning more sample efficient. As a result, the proposed approach shows strong performance in the few-shot learning domain. While it achieves superior results compared to state-of-the-art methods on multiple benchmarks in few-shot learning, it is comparable in the full-shot scenario. This study opens up avenues for efficient self-supervised learning methods that are more robust than current contrastive methods for sentence embedding.1 1 Introduction Measuring sentence similarity has been challenging due to the ambiguity and variability of linguistic ex- pressions. The community’s strong interest in the topic can be attributed to its applicability in numer- ous language processing applications, such as sen- timent analysis, information retrieval, and semantic search (Pilehvar and Navigli, 2015; Iyyer et al., 2015). Language models perform well on these tasks but typically require fine-tuning on the down- stream task and corpora (Reimers and Gurevych, 2019; Devlin et al., 2018; Pfeiffer et al., 2020; Mos- bach et al., 2021). In terms of sentence embed- dings, contrastive learning schemes have already been adopted successfully (van den Oord et al., 2018; Liu et al., 2021; Gao et al., 2021; Carlsson et al., 2021). The idea of contrastive learning is that positive and negative pairs are generated given 1Source code and pre-trained models are available at: https://github.com/SAP-samples/acl2023-micse/ a batch of samples. Whereas the positive pairs are obtained via augmentation, negative pairs are often created by random collation of sentences. Following the construction of pairs, contrastive learning forces the network to learn feature rep- resentations by pushing apart different samples (negative pairs) or pulling together similar ones (positive pairs). While some methods seek to op- timize for selecting “hard” negative for negative pair generation (Zhou et al., 2022a), others investi- gated better augmentation techniques for positive pair creation. In this regard, many methods have been proposed to create augmentations to boost representation learning. Standard approaches for the augmentation aim at input data level (a.k.a dis- crete augmentation), which comprises word level operations such as swapping, insertion, deletion, and substitution (Xie et al., 2017; Coulombe, 2018; Wei and Zou, 2019). In contrast to that, contin- uous augmentation operates at the representation level, comprising approaches like interpolation or “mixup” on the embedding space (Chen et al., 2020; Cheng et al., 2020; Guo et al., 2019). Most re- cently, augmentation was also proposed in a more continuous fashion operating in a parameter level via simple techniques such as drop-out (Gao et al., 2021; …

Content truncated to 500 words