SimLM: Pre-training with Representation Bottleneck for Dense Passage Retrieval

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 2244–2258 July 9-14, 2023 ©2023 Association for Computational Linguistics SIMLM: Pre-training with Representation Bottleneck for Dense Passage Retrieval Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao Linjun Yang, Daxin Jiang, Rangan Majumder, Furu Wei Microsoft Corporation {wangliang,nanya,xiaolhu,binxjia,yang.linjun,djiang,ranganm,fuwei}@microsoft.com Abstract In this paper, we propose SIMLM (Similarity matching with Language Model pre-training), a simple yet effective pre-training method for dense passage retrieval. It employs a simple bottleneck architecture that learns to compress the passage information into a dense vector through self-supervised pre-training. We use a replaced language modeling objective, which is inspired by ELECTRA (Clark et al., 2020), to improve the sample efficiency and reduce the mismatch of the input distribution between pre-training and fine-tuning. SIMLM only re- quires access to an unlabeled corpus and is more broadly applicable when there are no labeled data or queries. We conduct experi- ments on several large-scale passage retrieval datasets and show substantial improvements over strong baselines under various settings. Remarkably, SIMLM even outperforms multi- vector approaches such as ColBERTv2 (San- thanam et al., 2021) which incurs significantly more storage cost. Our code and model check- points are available at https://github.com/ microsoft/unilm/tree/master/simlm. 1 Introduction Passage retrieval is an important component in ap- plications like ad-hoc information retrieval, open- domain question answering (Karpukhin et al., 2020), retrieval-augmented generation (Lewis et al., 2020) and fact verification (Thorne et al., 2018). Sparse retrieval methods such as BM25 were the dominant approach for several decades, and still play a vital role nowadays. With the emer- gence of large-scale pre-trained language models (PLM) (Devlin et al., 2019), increasing attention is being paid to neural dense retrieval methods (Yates et al., 2021). Dense retrieval methods map both queries and passages into a low-dimensional vector space, where the relevance between the queries and passages are measured by the dot product or cosine similarity between their respective vectors. PLM MS-MARCO GLUE BERT 33.7 80.5 RoBERTa 33.1 88.1 ELECTRA 31.9 89.4 Table 1: Inconsistent performance trends between dif- ferent models on retrieval task and NLU tasks. We re- port MRR@10 on the dev set of MS-MARCO passage ranking dataset and test set results on GLUE bench- mark. Details are available in the Appendix A. Like other NLP tasks, dense retrieval benefits greatly from a strong general-purpose pre-trained language model. However, general-purpose pre- training does not solve all the problems. As shown in Table 1, improved pre-training techniques that are verified by benchmarks like GLUE (Wang et al., 2019) do not result in consistent performance gain for retrieval tasks. Similar observations are also made by Lu et al. (2021). We hypothesize that, to perform robust retrieval, the [CLS] vector used for computing matching scores should encode all the essential information in the passage. The next-sentence prediction (NSP) task in BERT intro- duces some supervision signals for the [CLS] token, while RoBERTa (Liu et al., 2019) and ELECTRA do not have such sequence-level tasks. In this paper, we propose …

Content truncated to 500 words