Towards Robust Neural Retrieval with Source Domain Synthetic Pre-Finetuning

Proceedings of the 29th International Conference on Computational Linguistics, pages 1065–1070 October 12–17, 2022. 1065 Towards Robust Neural Retrieval with Source Domain Synthetic Pre-Finetuning Revanth Gangi Reddy1, Vikas Yadav3∗, Md Arafat Sultan2, Martin Franz2, Vittorio Castelli3∗, Heng Ji1, Avirup Sil2 1University of Illinois at Urbana-Champaign 2IBM Research AI 3Amazon AWS AI {revanth3,hengji}@illinois.edu, arafat.sultan@ibm.com, {franzm,avi}@us.ibm.com, vittorca@amazon.com Abstract Research on neural IR has so far been focused primarily on standard supervised learning set- tings, where it outperforms traditional term matching baselines. Many practical use cases of such models, however, may involve previ- ously unseen target domains. In this paper, we propose to improve the out-of-domain gener- alization of Dense Passage Retrieval (DPR)— a popular choice for neural IR—through syn- thetic data augmentation only in the source do- main. We empirically show that pre-finetuning DPR with additional synthetic data in its source domain (Wikipedia), which we generate using a fine-tuned sequence-to-sequence generator1, can be a low-cost yet effective first step towards its generalization. Across five different test sets, our augmented model shows more robust performance than DPR in both in-domain and zero-shot out-of-domain evaluation. 1 Introduction Traditional approaches to information retrieval (IR) such as TF-IDF (Salton and McGill, 1986) and BM25 (Robertson and Zaragoza, 2009) rely on lexical matching for query-passage alignment. In contrast, neural IR encodes passages and questions into continuous vector representations, enabling deeper semantic matching. Modern neural IR sys- tems (Lee et al., 2019; Chang et al., 2019) based on pre-trained masked language models (MLM) (De- vlin et al., 2019) typically employ a dual encoder architecture (Bromley et al., 1993), where two sep- arate MLMs encode the question and the passage. Karpukhin et al. (2020) show that useful weak su- pervision for such systems can be derived from the related task of machine reading comprehen- sion (MRC) (Kwiatkowski et al., 2019; Joshi et al., 2017). Their Dense Passage Retrieval (DPR) model demonstrates state-of-the-art (SOTA) in-domain ∗Work done while at IBM Research AI 1Synthetic question generation code is available at: https://github.com/primeqa/primeqa/tree/main/primeqa/qg performance on multiple Wikipedia-based datasets (Kwiatkowski et al., 2019; Joshi et al., 2017; Berant et al., 2013; Baudiš and Šediv`y, 2015), outperform- ing both term matching baselines like BM25 and prior neural approaches, e.g., the Inverse Cloze Task (Lee et al., 2019) and latent learning of the re- triever during MLM pre-training (Guu et al., 2020). Despite its high in-domain utility, however, Reddy et al. (2021) show that DPR performance can drop significantly in novel test domains. They propose target domain synthetic data augmentation as a solution to this problem, which augments DPR with additional synthetic training data generated from target domain text. While this approach does indeed improve DPR scores in the new test domain, it has a key practical limitation: for every new tar- get domain, it requires generating a new synthetic training corpus and re-training the model. Here we ask if an augmentation approach that only operates once in the source domain, and does not require re-training every time a new test domain is encoun- tered, can …

Content truncated to 500 words