ED2LM: Encoder-Decoder to Language Model for Faster Document Re-ranking Inference

Findings of the Association for Computational Linguistics: ACL 2022, pages 3747 - 3758 May 22-27, 2022 c⃝2022 Association for Computational Linguistics ED2LM: Encoder-Decoder to Language Model for Faster Document Re-ranking Inference Kai Hui∗ Honglei Zhuang Tao Chen Zhen Qin Jing Lu Dara Bahri Ji Ma Jai Gupta Cicero Nogueira dos Santos Yi Tay Donald Metzler kaihuibj@google.com Google Research Abstract State-of-the-art neural models typically encode document-query pairs using cross- attention for re-ranking. To this end, models generally utilize an encoder-only (like BERT) paradigm or an encoder-decoder (like T5) approach. These paradigms, however, are not without flaws, i.e., running the model on all query-document pairs at inference-time incurs a significant computational cost. This paper proposes a new training and inference paradigm for re-ranking. We propose to finetune a pretrained encoder-decoder model using in the form of document to query generation. Subsequently, we show that this encoder-decoder architecture can be decomposed into a decoder-only language model during inference. This results in significant inference time speedups since the decoder-only architecture only needs to learn to interpret static encoder embeddings during inference. Our experiments show that this new paradigm achieves results that are comparable to the more expensive cross-attention ranking approaches while being up to 6.8X faster. We believe this work paves the way for more efficient neural rankers that leverage large pretrained models. 1 Introduction Leveraging transformer architecture to model the concatenation of a query-document pair is a well-established approach for document ranking (Nogueira et al., 2020). Today, modern neural methods for re-ranking are based on the encoder- only (e.g., BERT (Devlin et al., 2019)) or encoder- decoder (e.g., T5 (Raffel et al., 2020)) paradigm where query-document interactions are modeled by the encoder’s attention mechanism. Unfortu- nately, these paradigms are computationally pro- hibitive given that the model has to be run on all document-query pairs during inference. To this ∗Corresponding Author end, it is commonplace to use less powerful but computationally lightweight dual encoder models (Nogueira et al., 2019a; Karpukhin et al., 2020; Xiong et al., 2020; Qu et al., 2021; Gao et al., 2021) for first-pass retrieval and to only run the more ex- pensive re-ranker on a small subset of retrieved can- didates. Even with this setup, cross-attention-based re-ranking can still be expensive, especially when larger pretrained Transformer models are used. As such, this paper is primarily concerned with im- proving inference-time re-ranking efficiency while maintaining comparable effectiveness to existing cross-attention models. The novelty of this paper lies in a new paradigm for re-ranking that provides up to 6.8X speedup without any degradation in shallow-pool effective- ness. Concretely, we propose a new method for inference-time decomposition of encoder-decoder architectures into decoder-only language models. Given a pretrained sequence-to-sequence model, we finetune the encoder-decoder model using a document-to-query multi-task loss. At inference, we decompose the encoder-decoder architecture into a decoder-only language model (LM) that learns to interpret from a memory store of encoded document tokens representations using attention. The document-query pair score can be interpreted as the likelihood of generating the query given the …

Content truncated to 500 words