Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrieval

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 2843 - 2853 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrieval Luyu Gao and Jamie Callan Language Technologies Institute Carnegie Mellon University {luyug, callan}@cs.cmu.edu Abstract Recent research demonstrates the effective- ness of using fine-tuned language mod- els (LM) for dense retrieval. However, dense retrievers are hard to train, typically requiring heavily engineered fine-tuning pipelines to re- alize their full potential. In this paper, we iden- tify and address two underlying problems of dense retrievers: i) fragility to training data noise and ii) requiring large batches to robustly learn the embedding space. We use the re- cently proposed Condenser pre-training archi- tecture, which learns to condense information into the dense vector through LM pre-training. On top of it, we propose coCondenser, which adds an unsupervised corpus-level contrastive loss to warm up the passage embedding space. Experiments on MS-MARCO, Natural Ques- tion, and Trivia QA datasets show that coCon- denser removes the need for heavy data engi- neering such as augmentation, synthesis, or fil- tering, and the need for large batch training. It shows comparable performance to RocketQA, a state-of-the-art, heavily engineered system, using simple small batch fine-tuning.1 1 Introduction Building upon the advancements of pre-trained lan- guage models (LM; Devlin et al. (2019); Liu et al. (2019)), dense retrieval has become an effective paradigm for text retrieval (Lee et al., 2019; Chang et al., 2020; Karpukhin et al., 2020; Qu et al., 2021; Gao et al., 2021a; Zhan et al., 2022). Recent re- search has however found that fine-tuning dense retrievers to realize their capacity requires carefully designed fine-tuning techniques. Early works in- clude iterative negative mining (Xiong et al., 2021) and multi-vector representations (Luan et al., 2020). The recent RocketQA system (Qu et al., 2021) sig- nificantly improves the performance of a dense retriever by designing an optimized fine-tuning 1Our code is available at https://github.com/ luyug/Condenser. pipeline that includes i) denoising hard negatives, which corrects mislabeling, and ii) large batch train- ing. While this is very effective, the entire pipeline is very heavy in computation and not feasible for people who do not have tremendous hardware re- sources, like those in academia. In this paper, we ask, instead of directly using the pipeline, can we take the insights of RocketQA to perform language model pre-training such that the pre-trained model can be easily fine-tuned on any target query set. Concretely, we ask what the optimized training in RocketQA solves. We hypothesize that typi- cal LMs are sensitive to mislabeling, which can cause detrimental updates to the model weights. Denoising can effectively remove the bad samples and their updates. On the other hand, for most LMs, the CLS vectors are either trained with a simple task (Devlin et al., 2019) or not explicitly trained at all (Liu et al., 2019). These vectors are far from being able to form an embedding space of passages …

Content truncated to 500 words