Compressing Sentence Representation for Semantic Retrieval via Homomorphic Projective Distillation

Findings of the Association for Computational Linguistics: ACL 2022, pages 774 - 781 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Compressing Sentence Representation for Semantic Retrieval via Homomorphic Projective Distillation Xuandong Zhao†, Zhiguo Yu‡, Ming Wu‡, Lei Li† †UC Santa Barbara ‡Microsoft {xuandongzhao,leili}@cs.ucsb.edu {zhiguo.yu,mingwu}@microsoft.com Abstract How to learn highly compact yet effective sen- tence representation? Pre-trained language models have been effective in many NLP tasks. However, these models are often huge and pro- duce large sentence embeddings. Moreover, there is a big performance gap between large and small models. In this paper, we propose Homomorphic Projective Distillation (HPD) to learn compressed sentence embeddings. Our method augments a small Transformer encoder model with learnable projection layers to pro- duce compact representations while mimicking a large pre-trained language model to retain the sentence representation quality. We evalu- ate our method with different model sizes on both semantic textual similarity (STS) and se- mantic retrieval (SR) tasks. Experiments show that our method achieves 2.7-4.5 points perfor- mance gain on STS tasks compared with pre- vious best representations of the same size. In SR tasks, our method improves retrieval speed (8.2×) and memory usage (8.0×) compared with state-of-the-art large models. Our imple- mentation is available at https://github. com/XuandongZhao/HPD. 1 Introduction It is a fundamental problem to learn compact yet ef- fective sentence representations. Good representa- tions have wide applications in NLP, including web search (Palangi et al., 2016), question answering (Hao et al., 2019), knowledge inference (Wang and Kuo, 2020), and machine translation (Yang et al., 2020). Sentence embedding models take a sentence as the input and output a fixed-length continuous vector representation. Based on BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019), re- cent sentence embedding models such as Sentence- BERT (SBERT) (Reimers and Gurevych, 2019) and SimCSE (Gao et al., 2021), are fine-tuned on sentence pair scoring tasks to learn better sentence representations, which show much improvement Augmented Sentences NLI Sentences Data augmentation PCA Average Pooling Pre-trained Large Language Model Projection Layer Average Pooling Pre-trained Small Language Model Teacher Student Fixed Mean Square Loss high dim same low dim Figure 1: Overview of Homomorphic Projective Distillation (HPD). In contrast to the teacher model, which is a large pre-trained language model with a fixed PCA dimension reduction module, the student model is a smaller language model with a learnable projection layer. The mean square error of both dimension reduc- tion results is used to train the student model. in downstream tasks. However, these models are big in two aspects. 1) They contain hundreds of millions to billions of parameters, which requires large memory and powerful machine to serve in production; 2) Their resulting embeddings are high dimensional (e.g. 1024), requiring huge database to store and index, which cause high search latency. Therefore, it is challenging to directly use these large models in real-world applications with strict throughput/latency requirement and bounded hard- ware resources. Our work focuses on reducing both the model size and the representation size. There has been …

Content truncated to 500 words