GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrieval

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2345 - 2360 July 10-15, 2022 ©2022 Association for Computational Linguistics GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrieval Kexin Wang1, Nandan Thakur2∗, Nils Reimers3∗, Iryna Gurevych1 1 Ubiquitous Knowledge Processing Lab, Technical University of Darmstadt 2 University of Waterloo, 3 Hugging Face www.ukp.tu-darmstadt.de Abstract Dense retrieval approaches can overcome the lexical gap and lead to significantly improved search results. However, they require large amounts of training data which is not avail- able for most domains. As shown in previous work (Thakur et al., 2021b), the performance of dense retrievers severely degrades under a domain shift. This limits the usage of dense re- trieval approaches to only a few domains with large training datasets. In this paper, we propose the novel unsuper- vised domain adaptation method Generative Pseudo Labeling (GPL), which combines a query generator with pseudo labeling from a cross-encoder. On six representative domain- specialized datasets, we find the proposed GPL can outperform an out-of-the-box state-of-the- art dense retrieval approach by up to 9.3 points nDCG@10. GPL requires less (unlabeled) data from the target domain and is more robust in its training than previous methods. We further investigate the role of six recent pre-training methods in the scenario of domain adaptation for retrieval tasks, where only three could yield improved results. The best ap- proach, TSDAE (Wang et al., 2021) can be combined with GPL, yielding another average improvement of 1.4 points nDCG@10 across the six tasks. The code and the models are available 1. 1 Introduction Information Retrieval (IR) is a central component of many natural language applications. Tradition- ally, lexical methods (Robertson et al., 1994) have been used to search through text content. However, these methods suffer from the lexical gap (Berger et al., 2000) and are not able to recognize synonyms and distinguish between ambiguous words. ∗Contributions made while being employed at UKP Lab. 1https://github.com/UKPLab/gpl Recently, information retrieval methods based on dense vector spaces have become popular to address these challenges. These dense retrieval methods map queries and passages2 to a shared, dense vector space and retrieve relevant hits by nearest-neighbor search. Significant improvement over traditional approaches has been shown for various tasks (Karpukhin et al., 2020; Xiong et al., 2021). This method is also adapted increasingly by industry to enhance the search functionalities of various applications (Choi et al., 2020; Huang et al., 2020). However, as shown in Thakur et al. (2021b), dense retrieval methods require a large amount of training data to work well.3 Most importantly, dense retrieval methods are extremely sensitive to domain shifts: Models trained on MS MARCO perform rather poorly for questions for COVID-19 scientific literature (Wang et al., 2020; Voorhees et al., 2021). The MS MARCO dataset was created before COVID-19, hence, it does not include any COVID-19 related topics and models did not learn how to represent this topic well in a vector space. In this …

Content truncated to 500 words