Learning to Retrieve Passages without Supervision

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2687 - 2700 July 10-15, 2022 ©2022 Association for Computational Linguistics Learning to Retrieve Passages without Supervision Ori Ram Gal Shachaf Omer Levy Jonathan Berant Amir Globerson Blavatnik School of Computer Science, Tel Aviv University ori.ram@cs.tau.ac.il Abstract Dense retrievers for open-domain question an- swering (ODQA) have been shown to achieve impressive performance by training on large datasets of question-passage pairs. In this work we ask whether this dependence on la- beled data can be reduced via unsupervised pretraining that is geared towards ODQA. We show this is in fact possible, via a novel pre- training scheme designed for retrieval. Our “recurring span retrieval” approach uses recur- ring spans across passages in a document to create pseudo examples for contrastive learn- ing. Our pretraining scheme directly controls for term overlap across pseudo queries and rel- evant passages, thus allowing to model both lexical and semantic relations between them. The resulting model, named Spider, performs surprisingly well without any labeled training examples on a wide range of ODQA datasets. Specifically, it significantly outperforms all other pretrained baselines in a zero-shot set- ting, and is competitive with BM25, a strong sparse baseline. Moreover, a hybrid retriever over Spider and BM25 improves over both, and is often competitive with DPR models, which are trained on tens of thousands of ex- amples. Last, notable gains are observed when using Spider as an initialization for supervised training.1 1 Introduction State-of-the-art models for retrieval in open domain question answering are based on learning dense text representations (Lee et al., 2019; Karpukhin et al., 2020; Qu et al., 2021). However, such models rely on large datasets of question-passage pairs for train- ing. These datasets are expensive and sometimes even impractical to collect (e.g., for new languages or domains), and models trained on them often fail 1Our code and models are publicly available: https://github.com/oriram/spider, and: https://huggingface.co/tau/spider k Top-k retrieval accuracy (%) 0 20 40 60 80 100 0 20 40 60 80 100 ICT BM25 Spider (Ours) Spider (Ours) + BM25 DPR DPR + BM25 Figure 1: Top-k retrieval accuracy of various unsuper- vised methods (solid lines) on the test set of Natural Questions (NQ). DPR (dotted) is supervised (trained on NQ) and given for reference. to generalize to new question distributions (Sci- avolino et al., 2021; Reddy et al., 2021). The above difficulty motivates the development of retrieval models that do not rely on large an- notated training sets, but are instead trained only on unlabeled text. Indeed, self-supervision for re- trieval has gained considerable attention recently (Lee et al., 2019; Guu et al., 2020; Sachan et al., 2021a; Fan et al., 2021). However, when applied in a “zero-shot” manner, such models are still outper- formed by sparse retrievers like BM25 (Robertson and Zaragoza, 2009) and by supervised models (see Sachan et al. 2021a). Moreover, models like REALM (Guu et al., 2020) and MSS (Sachan et al., 2021a,b) …

Content truncated to 500 words