Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3982–3992, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 3982 Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks Nils Reimers and Iryna Gurevych Ubiquitous Knowledge Processing Lab (UKP-TUDA) Department of Computer Science, Technische Universit¨at Darmstadt www.ukp.tu-darmstadt.de Abstract BERT (Devlin et al., 2018) and RoBERTa (Liu et al., 2019) has set a new state-of-the-art performance on sentence-pair regression tasks like semantic textual similarity (STS). How- ever, it requires that both sentences are fed into the network, which causes a massive com- putational overhead: Finding the most sim- ilar pair in a collection of 10,000 sentences requires about 50 million inference computa- tions (~65 hours) with BERT. The construction of BERT makes it unsuitable for semantic sim- ilarity search as well as for unsupervised tasks like clustering. In this publication, we present Sentence-BERT (SBERT), a modification of the pretrained BERT network that use siamese and triplet net- work structures to derive semantically mean- ingful sentence embeddings that can be com- pared using cosine-similarity. This reduces the effort for finding the most similar pair from 65 hours with BERT / RoBERTa to about 5 sec- onds with SBERT, while maintaining the ac- curacy from BERT. We evaluate SBERT and SRoBERTa on com- mon STS tasks and transfer learning tasks, where it outperforms other state-of-the-art sentence embeddings methods.1 1 Introduction In this publication, we present Sentence-BERT (SBERT), a modification of the BERT network us- ing siamese and triplet networks that is able to derive semantically meaningful sentence embed- dings2. This enables BERT to be used for certain new tasks, which up-to-now were not applicable for BERT. These tasks include large-scale seman- 1Code available: https://github.com/UKPLab/ sentence-transformers 2With semantically meaningful we mean that semantically similar sentences are close in vector space. tic similarity comparison, clustering, and informa- tion retrieval via semantic search. BERT set new state-of-the-art performance on various sentence classification and sentence-pair regression tasks. BERT uses a cross-encoder: Two sentences are passed to the transformer network and the target value is predicted. However, this setup is unsuitable for various pair regression tasks due to too many possible combinations. Finding in a collection of n = 10 000 sentences the pair with the highest similarity requires with BERT n·(n−1)/2 = 49 995 000 inference computations. On a modern V100 GPU, this requires about 65 hours. Similar, finding which of the over 40 mil- lion existent questions of Quora is the most similar for a new question could be modeled as a pair-wise comparison with BERT, however, answering a sin- gle query would require over 50 hours. A common method to address clustering and se- mantic search is to map each sentence to a vec- tor space such that semantically similar sentences are close. Researchers have started to input indi- vidual sentences into BERT and to derive fixed- size sentence embeddings. The most commonly used approach is to average the BERT output layer …

Content truncated to 500 words