Improving Passage Retrieval with Zero-Shot Question Generation

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3781 - 3797 December 7-11, 2022 ©2022 Association for Computational Linguistics Improving Passage Retrieval with Zero-Shot Question Generation Devendra Singh Sachan1,2∗, Mike Lewis3, Mandar Joshi4, Armen Aghajanyan3, Wen-tau Yih3, Joelle Pineau1,2,3, Luke Zettlemoyer3,5 1McGill University; 2Mila - Quebec AI Institute 3Meta AI Research; 4Google Research; 5University of Washington sachande@mila.quebec, mandarj@google.com {mikelewis,armenag,scottyih,jpineau,lsz}@meta.com Abstract We propose a simple and effective re-ranking method for improving passage retrieval in open question answering. The re-ranker re-scores retrieved passages with a zero-shot question generation model, which uses a pre-trained lan- guage model to compute the probability of the input question conditioned on a retrieved pas- sage. This approach can be applied on top of any retrieval method (e.g. neural or keyword- based), does not require any domain- or task- specific training (and therefore is expected to generalize better to data distribution shifts), and provides rich cross-attention between query and passage (i.e. it must explain every token in the question). When evaluated on a number of open-domain retrieval datasets, our re-ranker improves strong unsupervised retrieval mod- els by 6%-18% absolute and strong supervised models by up to 12% in terms of top-20 passage retrieval accuracy. We also obtain new state- of-the-art results on full open-domain question answering by simply adding the new re-ranker to existing models with no further changes.1 1 Introduction Text retrieval is a core sub-task in many NLP prob- lems, for example, open-domain question answer- ing where a document must be retrieved and then read to answer an input query. Queries and docu- ments are typically embedded in a shared represen- tation space to enable efficient search, before using a task-specific model to perform a deeper, token- level document analysis (e.g. a document reader that selects an answer span). We show that adding a zero-shot re-ranker to the retrieval stage of such models leads to large gains in performance, by do- ing deep token-level analysis with no task-specific data or tuning. ∗This work was done during an internship at Meta AI. 1Our codebase including data and checkpoints is avail- able at: https://github.com/DevSinghSachan/ unsupervised-passage-reranking 60 70 80 Natural Questions TriviaQA Contriever (Unsupervised) DPR (Supervised) Re-ranked Contriever Top-20 Accuracy Figure 1: After UPR re-ranking of the Contriever’s (unsupervised) (Izacard et al., 2022) top-1000 pas- sages, we outperform strong supervised models like DPR (Karpukhin et al., 2020) on Natural Questions and TriviaQA datasets. We focus on open-domain question answering and introduce a re-ranker based on zero-shot ques- tion generation with a pre-trained language model. Our re-ranker, which we call Unsupervised Passage Re-ranker (UPR), re-scores the retrieved passages by computing the likelihood of the input question conditioned on a retrieved passage.2 This simple method enables task-independent cross-attention between query and passage that can be applied on top of any retrieval method (e.g. neural or keyword- based) and is highly effective in practice (Figure 1). In part, UPR is inspired by the traditional mod- els of query scoring with count-based language models (Zhai and Lafferty, 2001). …

Content truncated to 500 words