Precise Zero-Shot Dense Retrieval without Relevance Labels

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 1762–1777 July 9-14, 2023 ©2023 Association for Computational Linguistics Precise Zero-Shot Dense Retrieval without Relevance Labels Luyu Gao∗ † Xueguang Ma∗ ‡ Jimmy Lin‡ Jamie Callan† †Language Technologies Institute, Carnegie Mellon University ‡David R. Cheriton School of Computer Science, University of Waterloo {luyug, callan}@cs.cmu.edu, {x93ma, jimmylin}@uwaterloo.ca Abstract While dense retrieval has been shown to be effective and efficient across tasks and lan- guages, it remains difficult to create effective fully zero-shot dense retrieval systems when no relevance labels are available. In this paper, we recognize the difficulty of zero-shot learn- ing and encoding relevance. Instead, we pro- pose to pivot through Hypothetical Document Embeddings (HyDE). Given a query, HyDE first zero-shot prompts an instruction-following language model (e.g., InstructGPT) to generate a hypothetical document. The document cap- tures relevance patterns but is “fake” and may contain hallucinations. Then, an unsupervised contrastively learned encoder (e.g., Contriever) encodes the document into an embedding vec- tor. This vector identifies a neighborhood in the corpus embedding space, from which simi- lar real documents are retrieved based on vec- tor similarity. This second step grounds the generated document to the actual corpus, with the encoder’s dense bottleneck filtering out the hallucinations. Our experiments show that HyDE significantly outperforms the state-of- the-art unsupervised dense retriever Contriever and shows strong performance comparable to fine-tuned retrievers across various tasks (e.g. web search, QA, fact verification) and in non- English languages (e.g., sw, ko, ja, bn).1 1 Introduction Dense retrieval (Lee et al., 2019; Karpukhin et al., 2020), the method of retrieving documents using semantic embedding similarities, has been shown to be successful across tasks like web search, ques- tion answering, and fact verification. A variety of methods such as negative mining (Xiong et al., 2021; Qu et al., 2021), distillation (Qu et al., 2021; Lin et al., 2021b; Hofstätter et al., 2021), retrieval- specific pre-training (Izacard et al., 2021; Gao and ∗ Equal contribution. 1No models were trained or fine-tuned in writing this paper. Our open-source code is available at https://github.com/ texttron/hyde. Callan, 2021; Lu et al., 2021; Gao and Callan, 2022; Liu and Shao, 2022) and scaling (Ni et al., 2022) have been proposed to improve the effectiveness of supervised dense retrieval models. Nevertheless, zero-shot dense retrieval still re- mains difficult. Many recent works consider the alternative transfer learning setup, where the dense retrievers are trained on a high-resource dataset and then evaluated on queries from different domains. MS MARCO (Bajaj et al., 2016), a dataset with a large number of manually judged query-document pairs, is the most commonly used. As argued by Izacard et al. (2021), in practice, however, the ex- istence of such a large dataset cannot always be assumed. Furthermore, MS MARCO restricts com- mercial use and cannot be adopted in a variety of real-world search scenarios. In this paper, we aim to build effective fully zero-shot dense retrieval systems that require no relevance supervision, work out-of-box and …

Content truncated to 500 words