SDR: Efficient Neural Re-ranking using Succinct Document Representation

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 6624 - 6637 May 22-27, 2022 c⃝2022 Association for Computational Linguistics SDR: Efficient Neural Re-ranking using Succinct Document Representation Nachshon Cohen∗ Amazon nachshon@amazon.com Amit Portnoy∗ Ben-Gurion University† amitport@post.bgu.ac.il Besnik Fetahu Amazon besnikf@amazon.com Amir Ingber Pinecone Systems† ingber@pinecone.io Abstract BERT based ranking models have achieved superior performance on various information retrieval tasks. However, the large number of parameters and complex self-attention op- erations come at a significant latency over- head. To remedy this, recent works propose late-interaction architectures, which allow pre- computation of intermediate document repre- sentations, thus reducing latency. Nonetheless, having solved the immediate latency issue, these methods now introduce storage costs and network fetching latency, which limit their adoption in real-life production systems. In this work, we propose the Succinct Docu- ment Representation (SDR) scheme that com- putes highly compressed intermediate doc- ument representations, mitigating the stor- age/network issue. Our approach first reduces the dimension of token representations by en- coding them using a novel autoencoder archi- tecture that uses the document’s textual con- tent in both the encoding and decoding phases. After this token encoding step, we further re- duce the size of the document representations using modern quantization techniques. Evaluation on MSMARCO’s passage re- reranking task show that compared to exist- ing approaches using compressed document representations, our method is highly efficient, achieving 4x–11.6x higher compression rates for the same ranking quality. Similarly, on the TREC CAR dataset, we achieve 7.7x higher compression rate for the same ranking quality. 1 Introduction Information retrieval (IR) systems traditionally comprise of two stages: retrieval and ranking. Given a user query, the role of the retrieval stage is to quickly retrieve a set of candidate documents ∗Both authors contributed equally to the paper. †Work carried out while working at Amazon. 103 104 105 106 Document corpus size (MB) 0.340 0.345 0.350 0.355 0.360 0.365 0.370 0.375 0.380 Ranking quality (MRR@10) BERTSPLIT (uncompressed) SDR SDR (float16) Baseline Figure 1: MRR@10 performance vs. document cor- pus size tradeoff, measured on the MSMARCO-DEV dataset. BERTSPLIT is a distilled late-interaction model with reduced vector width and no compression (§ 4.2). For MRR@10 above 0.35, SDR is 4x–11.6x more effi- cient compared to the baseline. from a (very large) search index. Retrieval is typi- cally fast but not accurate enough; in order to im- prove the quality of the end result for the user, the candidate documents are re-ranked using a more accurate but computationally expensive algorithm. Neural approaches have achieved the state of the art ranking performance in IR applications (Yates et al., 2021). Transformer networks such as BERT (Devlin et al., 2019) consistently show better rank- ing effectiveness at the cost of a higher computa- tional cost and latency (Nogueira and Cho, 2019). To rank k documents, the ranker is called k times with an input of the form (query, document), where the query is the same, but the document is different. Several works (MacAvaney et al., …

Content truncated to 500 words