Document Ranking with a Pretrained Sequence-to-Sequence Model

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 708–718 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 708 Document Ranking with a Pretrained Sequence-to-Sequence Model Rodrigo Nogueira,∗ Zhiying Jiang,∗ Ronak Pradeep, and Jimmy Lin David R. Cheriton School of Computer Science University of Waterloo {rodrigo.nogueira, zhiying.jiang, rpradeep, jimmylin}@uwaterloo.ca Abstract This work proposes the use of a pretrained sequence-to-sequence model for document ranking. Our approach is fundamentally differ- ent from a commonly adopted classification- based formulation based on encoder-only pretrained transformer architectures such as BERT. We show how a sequence-to-sequence model can be trained to generate relevance la- bels as “target tokens”, and how the under- lying logits of these target tokens can be in- terpreted as relevance probabilities for rank- ing. Experimental results on the MS MARCO passage ranking task show that our rank- ing approach is superior to strong encoder- only models. On three other document re- trieval test collections, we demonstrate a zero- shot transfer-based approach that outperforms previous state-of-the-art models requiring in- domain cross-validation. Furthermore, we find that our approach significantly outperforms an encoder-only architecture in a data-poor set- ting. We investigate this observation in more detail by varying target tokens to probe the model’s use of latent knowledge. Surprisingly, we find that the choice of target tokens impacts effectiveness, even for words that are closely related semantically. This finding sheds some light on why our sequence-to-sequence formu- lation for document ranking is effective. Code and models are available at pygaggle.ai. 1 Introduction A simple, straightforward formulation of ranking is to convert the task into a classification problem, and then sort the candidate items to be ranked based on the probability that each item belongs to the desired class. Applied to the document ranking problem in information retrieval—where given a query, the system’s task is to return a ranked list of documents from a large corpus that maximizes some ranking ∗Equal contribution. metric such as average precision or nDCG—the simplest formulation is to deploy a classifier that estimates the probability each document belongs to the “relevant” class, and then sort all the candidates by these estimates. Deep transformer models pretrained with language modeling objectives, exemplified by BERT (Devlin et al., 2019), have proven highly effective in a variety of classification and sequence labeling tasks in NLP; Nogueira and Cho (2019) are the first to demonstrate their effectiveness in ranking tasks. Since it is impractical to apply infer- ence to every document in a corpus with respect to a query, these techniques are typically applied to rerank a list of candidates. In a typical end-to-end system, these candidates are taken from the results of a keyword search based on a “classic” IR scor- ing function such as BM25 (Robertson et al., 1994). This leads to the standard multi-stage pipeline ar- chitecture where first-stage retrieval is followed by reranking using one or more machine learning mod- els (Asadi and Lin, 2013; Nogueira et al., 2019a). This architecture underlies nearly all transformer- based approaches to document …

Content truncated to 500 words