CODER: An efficient framework for improving retrieval through COntextual Document Embedding Reranking

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10626 - 10644 December 7-11, 2022 ©2022 Association for Computational Linguistics CODER: An efficient framework for improving retrieval through COntextual Document Embedding Reranking George Zerveas1, Navid Rekabsaz2, Daniel Cohen1,3, Carsten Eickhoff1 1AI Lab, Brown University, USA {george_zerveas,carsten}@brown.edu 2Johannes Kepler University Linz, LIT AI Lab, Austria navid.rekabsaz@jku.at 3Dataminr, USA daniel.cohen@dataminr.com Abstract Contrastive learning has been the dominant ap- proach to training dense retrieval models. In this work, we investigate the impact of ranking context – an often overlooked aspect of learning dense retrieval models. In particular, we exam- ine the effect of its constituent parts: jointly scoring a large number of negatives per query, using retrieved (query-specific) instead of ran- dom negatives, and a fully list-wise loss. To incorporate these factors into training, we intro- duce Contextual Document Embedding Rerank- ing (CODER), a highly efficient retrieval frame- work. When reranking, it incurs only a negli- gible computational overhead on top of a first- stage method at run time (∼ 5 ms delay per query), allowing it to be easily combined with any state-of-the-art dual encoder method. Mod- els trained through CODER can also be used as stand-alone retrievers. Evaluating CODER in a large set of experiments on the MS MARCO and TripClick collections, we show that the contextual reranking of precomputed document embeddings leads to a significant improvement in retrieval performance. This improvement becomes even more pronounced when more relevance information per query is available, shown in the TripClick collection, where we establish new state-of-the-art results by a large margin. 1 Introduction Neural text retrieval models typically rely on a con- trastive training optimization that uses (query, posi- tive document, negative document) triplets as train- ing samples. This scheme is especially popular, as it is well-suited to the computational constraints of large transformer-based language models such as BERT (Devlin et al., 2019) and its variants for retrieval (Nogueira and Cho, 2020; Khattab and Zaharia, 2020; Zhan et al., 2020b,a). Referred to as pair-wise training, the model is asked to score the similarity between the query embedding and a ground truth relevant (positive) document embed- ding, higher than the one between the query and a negative document embedding. While such contrastive learning approaches are effective, by only considering pairs of positive and negative documents at a time, they (1) deviate from the target objective of comparing a query against many documents while discarding inter-document information, and (2) depart from core list-wise eval- uation metrics like nDCG (Calauzènes et al., 2012). Addressing the first shortcoming, recent works have employed “in-batch” negatives: given a batch containing (query, positive document) tuples, the negatives to a tuple are set as the known positive documents from other queries within that batch. (e.g. Karpukhin et al. (2020); Luan et al. (2021); Zhan et al. (2020b); Hofstätter et al. (2021); Qu et al. (2021)). Although this approach efficently in- creases the number of negatives to improve perfor- mance, the presence of hard negative samples1 is critical …

Content truncated to 500 words