LaPraDoR: Unsupervised Pretrained Dense Retriever for Zero-Shot Text Retrieval

Findings of the Association for Computational Linguistics: ACL 2022, pages 3557 - 3569 May 22-27, 2022 c⃝2022 Association for Computational Linguistics LaPraDoR: Unsupervised Pretrained Dense Retriever for Zero-Shot Text Retrieval Canwen Xu1∗, Daya Guo2∗, Nan Duan3, Julian McAuley1 1University of California, San Diego, 2Sun Yat-sen University, 3Microsoft Research Asia 1{cxu,jmcauley}@ucsd.edu, 2guody5@mail2.sysu.edu.cn, 3nanduan@microsoft.com Abstract In this paper, we propose LaPraDoR, a pre- trained dual-tower dense retriever that does not require any supervised data for training. Specifically, we first present Iterative Con- trastive Learning (ICoL) that iteratively trains the query and document encoders with a cache mechanism. ICoL not only enlarges the num- ber of negative instances but also keeps rep- resentations of cached examples in the same hidden space. We then propose Lexicon- Enhanced Dense Retrieval (LEDR) as a sim- ple yet effective way to enhance dense re- trieval with lexical matching. We evaluate LaPraDoR on the recently proposed BEIR benchmark, including 18 datasets of 9 zero- shot text retrieval tasks. Experimental results show that LaPraDoR achieves state-of-the-art performance compared with supervised dense retrieval models, and further analysis reveals the effectiveness of our training strategy and ob- jectives. Compared to re-ranking, our lexicon- enhanced approach can be run in milliseconds (22.5× faster) while achieving superior perfor- mance.1 1 Introduction Dense retrieval uses dense vectors to represent doc- uments and retrieve documents by similarity scores between query vectors and document vectors. Dif- ferent from cross-encoders (Reimers and Gurevych, 2019; Gao et al., 2020; MacAvaney et al., 2020) or late-interaction models (Khattab and Zaharia, 2020; Gao et al., 2021a), which predict a match score for each query-document pair thus are computationally costly, dense retrieval can be run in milliseconds, with the help of an approximate nearest neighbor (ANN) retrieval library, e.g., FAISS (Johnson et al., 2021). As a drawback, dense retrieval models of- ten require large supervised datasets like MS- ∗Equal contribution. 1Code and pretrained weights can be found at https: //github.com/JetRunner/LaPraDoR. MARCO (Nguyen et al., 2016) (533k training ex- amples) or NQ (Kwiatkowski et al., 2019) (133k training examples) for training. Unfortunately, Thakur et al. (2021) empirically show that models trained on one dataset suffer from an out-of-domain (OOD) problem when transferring to another. This hinders the applications of dense retrieval systems. On the other hand, creating a large supervised train- ing dataset for dense retrieval is time-consuming and expensive. For many low-resource languages, there is even no existing supervised dataset for re- trieval and it can be extremely difficult to construct one. The recently proposed BEIR benchmark (Thakur et al., 2021) highlights the generalization ability of text retrieval systems. The benchmark fea- tures a setting where models are trained on a large supervised dataset MS-MARCO (Nguyen et al., 2016) and then tested on 18 heterogeneous datasets of 9 tasks. In this paper, we propose Large-scale Pretrained Dense Zero-shot Retriever (LaPraDoR), a fully unsupervised pretrained re- triever for zero-shot text retrieval. While exist- ing dense retrievers need large supervised data and struggle to compete with a lexical matching approach like BM25 (Robertson …

Content truncated to 500 words