On the Sentence Embeddings from Pre-trained Language Models

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 9119–9130, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 9119 On the Sentence Embeddings from Pre-trained Language Models Bohan Li†,‡∗, Hao Zhou†, Junxian He‡, Mingxuan Wang†, Yiming Yang‡, Lei Li† †ByteDance AI Lab ‡Language Technologies Institute, Carnegie Mellon University {zhouhao.nlp,wangmingxuan.89,lileilab}@bytedance.com {bohanl1,junxianh,yiming}@cs.cmu.edu Abstract Pre-trained contextual representations like BERT have achieved great success in natu- ral language processing. However, the sen- tence embeddings from the pre-trained lan- guage models without fine-tuning have been found to poorly capture semantic meaning of sentences. In this paper, we argue that the se- mantic information in the BERT embeddings is not fully exploited. We first reveal the the- oretical connection between the masked lan- guage model pre-training objective and the se- mantic similarity task theoretically, and then analyze the BERT sentence embeddings em- pirically. We find that BERT always induces a non-smooth anisotropic semantic space of sentences, which harms its performance of semantic similarity. To address this issue, we propose to transform the anisotropic sen- tence embedding distribution to a smooth and isotropic Gaussian distribution through nor- malizing flows that are learned with an un- supervised objective. Experimental results show that our proposed BERT-flow method ob- tains significant performance gains over the state-of-the-art sentence embeddings on a va- riety of semantic textual similarity tasks. The code is available at https://github.com/ bohanli/BERT-flow. 1 Introduction Recently, pre-trained language models and its vari- ants (Radford et al., 2019; Devlin et al., 2019; Yang et al., 2019; Liu et al., 2019) like BERT (Devlin et al., 2019) have been widely used as represen- tations of natural language. Despite their great success on many NLP tasks through fine-tuning, the sentence embeddings from BERT without fine- tuning are significantly inferior in terms of se- mantic textual similarity (Reimers and Gurevych, ∗ The work was done when BL was an intern at ByteDance. 2019) – for example, they even underperform the GloVe (Pennington et al., 2014) embeddings which are not contextualized and trained with a much sim- pler model. Such issues hinder applying BERT sentence embeddings directly to many real-world scenarios where collecting labeled data is highly- costing or even intractable. In this paper, we aim to answer two major ques- tions: (1) why do the BERT-induced sentence em- beddings perform poorly to retrieve semantically similar sentences? Do they carry too little semantic information, or just because the semantic meanings in these embeddings are not exploited properly? (2) If the BERT embeddings capture enough semantic information that is hard to be directly utilized, how can we make it easier without external supervision? Towards this end, we first study the connection between the BERT pretraining objective and the se- mantic similarity task. Our analysis reveals that the sentence embeddings of BERT should be able to intuitively reflect the semantic similarity between sentences, which contradicts with experimental ob- servations. Inspired by Gao et al. (2019) who find that the language modeling performance can be limited by the learned anisotropic word embedding space …

Content truncated to 500 words