Generate, Discriminate and Contrast: A Semi-Supervised Sentence Representation Learning Framework

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8150 - 8161 December 7-11, 2022 ©2022 Association for Computational Linguistics Generate, Discriminate and Contrast: A Semi-Supervised Sentence Representation Learning Framework Yiming Chen† Yan Zhang† Bin Wang† Zuozhu Liu‡⋆∗ Haizhou Li♮,†,§ †National University of Singapore ‡Zhejiang University ⋆Angelalign Inc., China ♮The Chinese University of Hong Kong, Shenzhen, China §Kriston AI Lab, China yiming.chen@u.nus.edu, zuozhuliu@intl.zju.edu.cn {haizhou.li,eleyanz,bin.wang}@nus.edu.sg Abstract Most sentence embedding techniques heavily rely on expensive human-annotated sentence pairs as the supervised signals. Despite the use of large-scale unlabeled data, the perfor- mance of unsupervised methods typically lags far behind that of the supervised counterparts in most downstream tasks. In this work, we propose a semi-supervised sentence embed- ding framework, GenSE, that effectively lever- ages large-scale unlabeled data. Our method include three parts: 1) Generate: A genera- tor/discriminator model is jointly trained to synthesize sentence pairs from open-domain unlabeled corpus; 2) Discriminate: Noisy sen- tence pairs are filtered out by the discriminator to acquire high-quality positive and negative sentence pairs; 3) Contrast: A prompt-based contrastive approach is presented for sentence representation learning with both annotated and synthesized data. Comprehensive experiments show that GenSE achieves an average correla- tion score of 85.19 on the STS datasets and con- sistent performance improvement on four do- main adaptation tasks, significantly surpassing the state-of-the-art methods and convincingly corroborating its effectiveness and generaliza- tion ability.1 1 Introduction Sentence representation learning has recently at- tracted outspread research attention. It learns vec- tor representations for sentences, which can be sub- sequently utilized on a wide range of downstream tasks, including information retrieval (Thakur et al., 2021b; Wang et al., 2022; Misra et al., 2016), lan- guage understanding and evaluation (Cer et al., 2018; Conneau and Kiela, 2018; Perone et al., 2018; Zhang et al., 2021a). For sentence repre- sentation learning, the contrastive learning-based approaches, including supervised and unsupervised ∗ Corresponding author. 1Code, Synthetic data and Models available at https: //github.com/MatthewCYM/GenSE ones, have demonstrated to be the most efficient and effective (Zhang et al., 2020; Carlsson et al., 2021; Giorgi et al., 2021; Gao et al., 2021). In contrastive learning, the quality of positive and negative pairs has a large impact on the over- all performance (Chen et al., 2020; Gao et al., 2021). In particular, previous supervised con- trastive methods usually construct sentence pairs using human-annotated natural language inference (NLI) data (Bowman et al., 2015; Williams et al., 2018), and outperform unsupervised approaches by a large margin (Gao et al., 2021). However, the state-of-the-art supervised methods usually rely on multi-source labeled data to generalize to various downstream tasks (Reimers and Gurevych, 2019), while such large-scale annotated data across do- mains are not always available. Recent works also attempt to leverage the re- sources of unlabeled sentences for better sen- tence representation learning. A straightforward choice is to adopt unsupervised or self-supervised paradigms, such as BERT-flow (Li et al., 2020), SimCSE (Gao et al., 2021). However, the perfor- mance of these methods is still far behind the …

Content truncated to 500 words