Questions Are All You Need to Train a Dense Passage Retriever

Questions Are All You Need to Train a Dense Passage Retriever Devendra Singh Sachan1,2, Mike Lewis3, Dani Yogatama4, Luke Zettlemoyer3,5, Joelle Pineau1,2,3, Manzil Zaheer4 1McGill University, Canada; 2Mila - Quebec AI Institute, Canada; 3Meta AI, USA; 4Google DeepMind, USA; 5University of Washington, USA sachande@mila.quebec, {dyogatama, manzilzaheer}@google.com {mikelewis,lsz,jpineau}@meta.com Abstract We introduce ART, a new corpus-level autoen- coding approach for training dense retrieval models that does not require any labeled train- ing data. Dense retrieval is a central challenge for open-domain tasks, such as Open QA, where state-of-the-art methods typically re- quire large supervised datasets with custom hard-negative mining and denoising of positive examples. ART, in contrast, only requires ac- cess to unpaired inputs and outputs (e.g., ques- tions and potential answer passages). It uses a new passage-retrieval autoencoding scheme, where (1) an input question is used to retrieve a set of evidence passages, and (2) the pas- sages are then used to compute the probability of reconstructing the original question. Train- ing for retrieval based on question reconstruc- tion enables effective unsupervised learning of both passage and question encoders, which can be later incorporated into complete Open QA systems without any further finetuning. Extensive experiments demonstrate that ART obtains state-of-the-art results on multiple QA retrieval benchmarks with only generic ini- tialization from a pre-trained language model, removing the need for labeled data and task- specific losses.1 1 Introduction Dense passage retrieval methods (Karpukhin et al., 2020; Xiong et al., 2021), initialized with encoders such as BERT (Devlin et al., 2019) and trained using supervised contrastive losses (Oord et al., 2018), have surpassed the performance achieved by previously popular keyword-based approaches like BM25 (Robertson and Zaragoza, 2009). Such retrievers are core components in models for open-domain tasks, such as Open QA, where state-of-the-art methods typically require 1Our code and model checkpoints are available at: https://github.com/DevSinghSachan/art. large supervised datasets with custom hard- negative mining and denoising of positive ex- amples. In this paper, we introduce the first unsupervised method, based on a new corpus-level autoencoding approach, that can match or sur- pass strong supervised performance levels with no labeled training data or task-specific losses. We propose ART, Autoencoding-based Re- triever Training, which only assumes access to sets of unpaired questions and passages. Given an input question, ART first retrieves a small set of possible evidence passages. It then reconstructs the original question by attending to these pas- sages (see Figure 1 for an overview). The key idea in ART is to consider the retrieved passages as a noisy representation of the original question and question reconstruction probability as a way of denoising that provides soft-labels for how likely each passage is to have been the correct result. To bootstrap the training of a strong model, it is important to both have a strong initial re- trieval model and to be able to compute reliable initial estimates of question reconstruction proba- bility when conditioned on a (retrieved) passage. Although passage representations from BERT- style models are known to be reasonable retrieval baselines, it is …

Content truncated to 500 words