Sentence Representation Learning with Generative Objective rather than Contrastive Objective

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3356 - 3368 December 7-11, 2022 ©2022 Association for Computational Linguistics Sentence Representation Learning with Generative Objective rather than Contrastive Objective Bohong Wu1,2, Hai Zhao1,2,∗ 1 Department of Computer Science and Engineering, Shanghai Jiao Tong University 2 Key Laboratory of Shanghai Education Commission for Intelligent Interaction and Cognitive Engineering, Shanghai Jiao Tong University chengzhipanpan@sjtu.edu.cn,zhaohai@cs.sjtu.edu.cn Abstract Though offering amazing contextualized token- level representations, current pre-trained lan- guage models take less attention on accurately acquiring sentence-level representation dur- ing their self-supervised pre-training. How- ever, contrastive objectives which dominate the current sentence representation learning bring little linguistic interpretability and no performance guarantee on downstream seman- tic tasks. We instead propose a novel genera- tive self-supervised learning objective based on phrase reconstruction. To overcome the drawbacks of previous generative methods, we carefully model intra-sentence structure by breaking down one sentence into pieces of important phrases. Empirical studies show that our generative learning achieves powerful enough performance improvement and outper- forms the current state-of-the-art contrastive methods not only on the STS benchmarks, but also on downstream semantic retrieval and reranking tasks. Our code is available at https://github.com/chengzhipanpan/PaSeR. 1 Introduction Sentence Representation Learning has long been a hot research topic (Conneau et al., 2017; Cer et al., 2018), for its effectiveness in a variety of downstream tasks like information retrieval and question answering (Yang et al., 2018). Although pre-trained language models (PrLMs) like BERT (Devlin et al., 2019) have achieved overwhelming performance on various token-level tasks, they are also criticized for being unable to produce high-quality sentence-level represen- tations. Research (Li et al., 2020) has shown that the native sentence representation produced by the “[CLS]” token of BERT shows extremely poor per- formance on sentence evaluation benchmarks like semantic textual similarity (STS) tasks. ∗*Corresponding author. This work was supported in part by the Key Projects of National Natural Science Foundation of China under Grants U1836222 and 61733011. The primary cause of these low-quality sen- tence representations is the lack of effective self- supervised sentence-level training objectives. As discussed in ConSERT (Yan et al., 2021), the original sentence-level pretraining objective Next Sentence Prediction (NSP) is too weak to pro- vide high-quality sentence representation. There- fore, recent researchers are seeking other effective self-supervised sentence-level objectives (Carlsson et al., 2020; Yan et al., 2021; Gao et al., 2021). Generally, self-supervised methods include both (i) generative methods, like Masked Language Mod- elling (MLM), and (ii) contrastive methods, like Next Sentence Prediction (NSP). By treating one sentence as a whole and con- trasting sentence representations with each other in the same training batch, contrastive methods have been shown extremely effective in Sentence Rep- resentation Learning in recent years. Generally, contrastive methods often use various data augmen- tation techniques to create different views for one sentence, and align the representations of these views within the same batch. ConSERT (Yan et al., 2021) utilizes techniques including token shuffling, feature cutoff, etc., and provides a general con- trastive learning framework for …

Content truncated to 500 words