PromptBERT: Improving BERT Sentence Embeddings with Prompts

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8826 - 8837 December 7-11, 2022 ©2022 Association for Computational Linguistics PromptBERT: Improving BERT Sentence Embeddings with Prompts Ting Jiang1,∗, Jian Jiao3, Shaohan Huang3, Zihan Zhang3, Deqing Wang1,4,†, Fuzhen Zhuang1,2,4 , Furu Wei3, Haizhen Huang3, Denvy Deng3 , Qi Zhang3 1SKLSDE Lab, School of Computer, Beihang University, Beijing, China 2Institute of Artificial Intelligence, Beihang University, Beijing, China 3 Microsoft 4 Zhongguancun Laboratory, Beijing, China {royokong, dqwang, zhuangfuzhen}@buaa.edu.cn {shaohanh, zihzha, fuwei, hhuang, liazha, zhang.qi}@microsoft.com Abstract We propose PromptBERT, a novel contrastive learning method for learning better sentence representation. We firstly analyze the drawback of current sentence embedding from original BERT and find that it is mainly due to the static token embedding bias and ineffective BERT layers. Then we propose the first prompt-based sentence embeddings method and discuss two prompt representing methods and three prompt searching methods to make BERT achieve bet- ter sentence embeddings. Moreover, we pro- pose a novel unsupervised training objective by the technology of template denoising, which substantially shortens the performance gap be- tween the supervised and unsupervised set- tings. Extensive experiments show the effec- tiveness of our method. Compared to SimCSE, PromptBert achieves 2.29 and 2.58 points of improvement based on BERT and RoBERTa in the unsupervised setting. Our code is avail- able at https://github.com/kongds/ Prompt-BERT. 1 Introduction In recent years, we have witnessed the success of pre-trained language models like BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) in sentence embeddings (Gao et al., 2021b; Yan et al., 2021). However, original BERT still shows poor performance in sentence embeddings (Reimers and Gurevych, 2019; Li et al., 2020). The most com- monly used example is that it underperforms the traditional word embedding methods like GloVe (Pennington et al., 2014). Previous research has linked anisotropy to ex- plain the poor performance of original BERT (Li et al., 2020; Yan et al., 2021; Gao et al., 2021b). Anisotropy makes the token embeddings occupy a narrow cone, resulting in a high similarity between * Work done during internship at microsoft. † Corresponding Author. any sentence pair (Li et al., 2020). Li et al. (2020) proposed a normalizing flows method to trans- form the sentence embeddings distribution to a smooth and isotropic Gaussian distribution and Yan et al. (2021) presented a contrastive framework to transfer sentence representation. The goal of these methods is to eliminate anisotropy in sentence em- beddings. However, we find that anisotropy may not be the primary cause of poor semantic similar- ity. For example, averaging the last layer of original BERT is even worse than averaging its static token embeddings in semantic textual similarity task, but the sentence embeddings from last layer are less anisotropic than static token embeddings. Following this result, we find original BERT layers actually damage the quality of sentence em- beddings. However, if we treat static token embed- dings as word embedding, it still yields unsatisfac- tory results compared to GloVe. Inspired by (Li et al., …

Content truncated to 500 words