Multi-Vector Attention Models for Deep Re-ranking

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5452–5456 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 5452 Multi-Vector Attention Models for Deep Re-ranking Giulio Zhou∗ Carnegie Mellon University giuliozhou@cmu.edu Jacob Devlin Google Research jacobdevlin@google.com Abstract Large-scale document retrieval systems often utilize two styles of neural network models which live at two different ends of the joint computation vs. accuracy spectrum. The first style is dual encoder (or two-tower) mod- els, where the query and document representa- tions are computed completely independently and combined with a simple dot product op- eration. The second style is cross-attention models, where the query and document fea- tures are concatenated in the input layer and all computation is based on the joint query- document representation. Dual encoder mod- els are typically used for retrieval and deep re-ranking, while cross-attention models are typically used for shallow re-ranking. In this paper, we present a lightweight architec- ture that explores this joint cost vs. accu- racy trade-off based on multi-vector attention (MVA). We thoroughly evaluate our method on the MS-MARCO passage retrieval dataset and show how to efficiently trade off retrieval accuracy with joint computation and offline document storage cost. We show that a highly compressed document representation and in- expensive joint computation can be achieved through a combination of learned pooling to- kens and aggressive downprojection. Our code and model checkpoints are available on GitHub. 1 Introduction Classical information retrieval systems used weighted sparse keyword matching to retrieve rel- evant documents for incoming search queries. A common approach has been to re-rank these docu- ments with a neural network model which takes the concatenation of the query text and document text as input, and emits a relevance score. We refer to this as a cross-attention network, depicted in Fig- ure 1a. In modern NLP, these systems are typically ∗Work done while at Google. pre-trained using a technique such as BERT (De- vlin et al., 2019) and then fine-tuned on human- labeled relevance data (Han et al., 2020; Ding et al., 2020; Nogueira et al., 2019). However, even if the re-ranking model is a state- of-the-art neural system, it is still limited by the documents that were produced by the retrieval sys- tem. Because of this, end-to-end neural approaches to retrieval have become popular to improve the rel- evance of the documents produced in the retrieval stage. These models typically take the form of a dual encoder network (also called a “two-tower net- work,” “Siamese network,” and “DSSM” (Huang et al., 2013), as depicted in Figure 1b), which emits a “query vector” and “document vector” condi- tioned on the query text and document text respec- tively. The relevance score is typically defined as the dot product (or cosine distance) between these vectors. With a dual encoder model, each document vec- tor in the corpus can be precomputed offline, and the query vector only needs to be computed once per incoming query. Since scoring each query- document pair is a simple dot product, …

Content truncated to 500 words