Zero-Shot Dense Retrieval with Momentum Adversarial Domain Invariant Representations

Findings of the Association for Computational Linguistics: ACL 2022, pages 4008 - 4020 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Zero-Shot Dense Retrieval with Momentum Adversarial Domain Invariant Representations Ji Xin1∗, Chenyan Xiong2, Ashwin Srinivasan2, Ankita Sharma2, Damien Jose2, Paul N. Bennett2 1 University of Waterloo 2 Microsoft ji.xin@uwaterloo.ca chenyan.xiong,ashwinsr,ankita.sharma, dajose,paul.n.bennett@microsoft.com Abstract Dense retrieval (DR) methods conduct text re- trieval by first encoding texts in the embedding space and then matching them by nearest neigh- bor search. This requires strong locality proper- ties from the representation space, e.g., close al- locations of each small group of relevant texts, which are hard to generalize to domains with- out sufficient training data. In this paper, we aim to improve the generalization ability of DR models from source training domains with rich supervision signals to target domains without any relevance label, in the zero-shot setting. To achieve that, we propose Momentum adver- sarial Domain Invariant Representation learn- ing (MoDIR), which introduces a momentum method to train a domain classifier that dis- tinguishes source versus target domains, and then adversarially updates the DR encoder to learn domain invariant representations. Our experiments show that MoDIR robustly outper- forms its baselines on 10+ ranking datasets col- lected in the BEIR benchmark in the zero-shot setup, with more than 10% relative gains on datasets with enough sensitivity for DR mod- els’ evaluation. Source code is available at https://github.com/ji-xin/modir. 1 Introduction Rather than matching texts in the bag-of-words space, Dense Retrieval (DR) methods first encode texts into a dense embedding space (Lee et al., 2019; Karpukhin et al., 2020; Xiong et al., 2021) and then conduct text retrieval using efficient nearest neigh- bor search (Chen et al., 2018; Guo et al., 2020; Johnson et al., 2021). With pre-trained language models and dedicated fine-tuning techniques, the learned representation space has significantly ad- vanced the first stage retrieval accuracy of many lan- guage systems, including web search (Xiong et al., ∗Work partly done during Ji’s internship at Microsoft. Source Positive Source Negative Target Positive Target Negative Source Query Target Query Figure 1: T-SNE plots of embedding space of a BERT reranker for q–d pairs (left) and ANCE dense retriever for queries/documents (right). Both models are trained on web search and transferred to medical search. 2021), grounded generation (Lewis et al., 2020), open domain question answering (Karpukhin et al., 2020; Izacard and Grave, 2020), etc. Purely using the learned embedding space for retrieval has raised concerns on the generalization ability, especially in scenarios without dedicated su- pervision signals. Many have observed diminishing advantages of DR models in various datasets if they are not fine-tuned with task-specific labels, i.e., in the zero-shot setup (Thakur et al., 2021). However, in many scenarios outside commercial web search, zero-shot is the norm. Obtaining training labels is difficult, expensive, and sometimes infeasible, especially in special domains (e.g., medical) where annotation requires strong expertise or is even pro- hibited because of privacy constraints. The lack of zero-shot ability hinders the democratization of advancements in dense retrieval …

Content truncated to 500 words