CoRT: Complementary Rankings from Transformers

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4194–4204 June 6–11, 2021. ©2021 Association for Computational Linguistics 4194 CoRT: Complementary Rankings from Transformers Marco Wrzalik Dirk Krechel RheinMain University of Applied Sciences, Germany {firstname.lastname}@hs-rm.de Abstract Many recent approaches towards neural infor- mation retrieval mitigate their computational costs by using a multi-stage ranking pipeline. In the first stage, a number of potentially rel- evant candidates are retrieved using an effi- cient retrieval model such as BM25. Although BM25 has proven decent performance as a first-stage ranker, it tends to miss relevant pas- sages. In this context we propose CoRT, a simple neural first-stage ranking model that leverages contextual representations from pre- trained language models such as BERT to com- plement term-based ranking functions while causing no significant delay at query time. Us- ing the MS MARCO dataset, we show that CoRT significantly increases the candidate re- call by complementing BM25 with missing candidates. Consequently, we find subsequent re-rankers achieve superior results with less candidates. We further demonstrate that pas- sage retrieval using CoRT can be realized with surprisingly low latencies. 1 Introduction The successful development of neural ranking models over the past few years has rapidly ad- vanced state-of-the-art performance in information retrieval (Guo et al., 2020; Craswell et al., 2020). One key aspect of the success is the exploitation of query-document interactions based on token repre- sentations from self-supervised language models (LMs) (Devlin et al., 2019; Radford et al., 2019; Pennington et al., 2014). Due to high computa- tional effort, however, these interaction-focused approaches are limited to re-ranking scenarios and thus they depend on the effectiveness of first-stage ranking models for candidate retrieval. Term-based retrieval models such as BM25 have proven decent performance in this task, but tend to miss relevant passages. In this context, we propose COmple- mentary Rankings from Transformers (CoRT), a simple neural first-stage ranking model that lever- ages contextual representations from transformer- based language models (Vaswani et al., 2017; De- vlin et al., 2019) to complement term-based first- stage rankings. CoRT optimizes an underlying text encoder towards representations that reflect relevance through vector similarity. The model is trained to act complementary to term-based re- trieval by using passages from BM25 rankings as negative examples. We study the characteristics of CoRT with four types of experiments based on the MS MARCO dataset. First, we measure vari- ous ranking metrics and compare the results with first-stage ranking baselines and competitors. In course of this, we demonstrate the portion of rel- evant candidates that are added by CoRT. Sec- ond, we combine the candidates from CoRT and BM25 with a state-of-the-art re-ranker based on BERT (Nogueira and Cho, 2019) and investigate how many candidates are needed to saturate the ranking quality. Third, we train CoRT with var- ious representation sizes and measure its impact on the first-stage ranking quality. Fourth, we mea- sure the retrieval latencies of CoRT with two re- trieval modalities: a distributed exhaustive search …

Content truncated to 500 words