Dimension Reduction for Efficient Dense Retrieval via Conditional Autoencoder

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5692 - 5698 December 7-11, 2022 ©2022 Association for Computational Linguistics Dimension Reduction for Efficient Dense Retrieval via Conditional Autoencoder Zhenghao Liu1, Han Zhang1, Chenyan Xiong2, Zhiyuan Liu3, Yu Gu1, and Xiaohua Li1 1Department of Computer Science and Technology, Northeastern University, China 2Microsoft Research, United States 3Department of Computer Science and Technology, Institute for AI, Tsinghua University, China Beijing National Research Center for Information Science and Technology, China {liuzhenghao,guyu,lixiaohua}@mail.neu.edu.cn zhanghan@stumail.neu.edu.cn; chenyan.xiong@microsoft.com liuzy@tsinghua.edu.cn Abstract Dense retrievers encode queries and documents and map them in an embedding space using pre-trained language models. These embed- dings need to be high-dimensional to fit train- ing signals and guarantee the retrieval effective- ness of dense retrievers. However, these high- dimensional embeddings lead to larger index storage and higher retrieval latency. To reduce the embedding dimensions of dense retrieval, this paper proposes a Conditional Autoencoder (ConAE) to compress the high-dimensional em- beddings to maintain the same embedding dis- tribution and better recover the ranking fea- tures. Our experiments show that ConAE is effective in compressing embeddings by achiev- ing comparable ranking performance with its teacher model and making the retrieval system more efficient. Our further analyses show that ConAE can alleviate the redundancy of the em- beddings of dense retrieval with only one linear layer. All codes of this work are available at https://github.com/NEUIR/ConAE. 1 Introduction As the first stage of numerous multi-stage IR and NLP tasks (Nogueira et al., 2019; Chen et al., 2017; Thorne et al., 2018), dense retrievers (Xiong et al., 2021a) have shown lots of advances in conduct- ing semantic searching and avoiding the vocabu- lary mismatch problem (Robertson and Zaragoza, 2009). Dense retrievers usually encode queries and documents as high-dimensional embeddings, which are necessary to guarantee retrieval effective- ness during training (Ma et al., 2021; Reimers and Gurevych, 2021). Nevertheless, high dimensional embeddings usually exhaust the memory to store the index and lead to longer retrieval latency (Indyk and Motwani, 1998; Meiser, 1993). The research of building efficient dense retrieval systems has been stimulated recently (Min et al., 2021). To reduce the dimensions of document embeddings, existing work reserves the principle dimensions or compresses query and document embeddings for building more efficient retriev- ers (Yang and Seo, 2021; Ma et al., 2021). There are two challenges in compressing embed- dings of dense retrievers: The compressed embed- dings should share a similar distribution with the original embeddings, making the low-dimensional embedding space uniform and the document em- beddings distinguishable; All the compressed em- beddings should have the ability to maintain the maximal information for matching related queries and documents during retrieval, which helps better align the related query-document pairs. This paper proposes a Conditional Autoencoder (ConAE), which aims to build efficient dense re- trieval systems by reducing the embedding dimen- sions of queries and documents. ConAE first en- codes high-dimensional embeddings into a low- dimensional embedding space and then gener- ates embeddings that can be aligned to related queries …

Content truncated to 500 words