Multi-stage Training with Improved Negative Contrast for Neural Passage Retrieval

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6091–6103 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 6091 Multi-stage Training with Improved Negative Contrast for Neural Passage Retrieval Jing Lu∗, Gustavo Hernández Ábrego, Ji Ma, Jianmo Ni, Yinfei Yang Google Research {ljwinnie, gustavoha, maji, jianmon, yinfeiy}@google.com Abstract In the context of neural passage retrieval, we study three promising techniques: synthetic data generation, negative sampling and fusion. We systematically investigate how these tech- niques contribute to the performance of the retrieval system and how they complement each other. We propose a multi-stage frame- work comprising of pre-training with synthetic data, fine-tuning with labeled data and nega- tive sampling at both stages. We study six negative sampling strategies and apply them to the fine-tuning stage and, as a noteworthy novelty, to the synthetic data that we use for pre-training. Also, we explore fusion methods that combine negatives from different strate- gies. We evaluate our system using two pas- sage retrieval tasks for open-domain QA and using MS MARCO. Our experiments show that augmenting the negative contrast in both stages is effective to improve passage retrieval accuracy and, importantly, they also show that synthetic data generation and negative sam- pling have additive benefits. Moreover, using fusion of different kinds allows us to reach per- formance that establishs a new state-of-the-art level in two of the tasks we evaluated. 1 Introduction Recently, there is a surge of interest in neural first- stage retrieval models (Yang et al., 2020; Guo et al., 2021). These models overcome the lexical gap is- sue of traditional models based on term matching (Robertson and Zaragoza, 2009) by projecting both query and document to a shared dense space. Find- ing relevant documents can then be achieved by employing nearest neighbor search. Neural first- stage retrieval models have shown competitive on many benchmark data sets (Karpukhin et al., 2020; Xiong et al., 2021; Qu et al., 2021), and combin- ing them with term matching-based models further boosts their retrieval performance (Bendersky et al., 2020). ∗Work done during an internship at Google. Arguably, abundance of training data and neg- ative sampling strategies have been the two most important factors to the success of neural retrieval models. On one hand, deep Neural Networks are data hungry due to their vast volume of model pa- rameters. Ma et al. (2021) has shown that synthetic question generation can be effective to mitigate the data scarcity issue in low-resource settings. In this work we are interested in exploring how synthetic question generation can further improve the neu- ral retrieval models when there is already a decent amount of supervised data available. We propose a two-stage training strategy where we first train the dense retrieval model on synthetic question- passage pairs and then, as illustrated in Fig 1, we fine-tune it on supervised data. We show that such methodology substantially improves upon baseline models. On the other hand, previous works (Karpukhin et al., 2020; Xiong et al., 2021) found that utilizing extra …

Content truncated to 500 words