DPTDR: Deep Prompt Tuning for Dense Passage Retrieval

Proceedings of the 29th International Conference on Computational Linguistics, pages 1193–1202 October 12–17, 2022. 1193 DPTDR: Deep Prompt Tuning for Dense Passage Retrieval Zhengyang Tang1,2, Benyou Wang2, and Ting Yao1 1Tencent Inc. 2The Chinese University of Hong Kong, Shenzhen {zhytang,tessieyao}@tencent.com, wangbenyou@cuhk.edu.cn Abstract Deep prompt tuning (DPT) has gained great success in most natural language process- ing (NLP) tasks. However, it is not well- investigated in dense retrieval where fine- tuning (FT) still dominates. When deploying multiple retrieval tasks using the same back- bone model (e.g., RoBERTa), FT-based meth- ods are unfriendly in terms of deployment cost: each new retrieval model needs to repeatedly deploy the backbone model without reuse. To reduce the deployment cost in such a scenario, this work investigates applying DPT in dense retrieval. The challenge is that directly apply- ing DPT in dense retrieval largely underper- forms FT methods. To compensate for the per- formance drop, we propose two model-agnostic and task-agnostic strategies for DPT-based re- trievers, namely retrieval-oriented intermediate pretraining and unified negative mining, as a general approach that could be compatible with any pre-trained language model and retrieval task. The experimental results 1 show that the proposed method (called DPTDR) outperforms previous state-of-the-art models on both MS- MARCO and Natural Questions. We also con- duct ablation studies to examine the effective- ness of each strategy in DPTDR. We believe this work facilitates the industry, as it saves enormous efforts and costs of deployment and increases the utility of computing resources. 1 Introduction Fine-tuning (FT) has been a de facto approach for effective dense passage retrieval (Karpukhin et al., 2020; Xiong et al., 2020) based on pre-trained lan- guage models (PLM). However, FT is unfriendly for industrial deployment in multi-task scenarios. Imaging for cloud service providers or infrastruc- ture teams of search companies, each retrieval model (w.r.t., an individual task) necessarily re- deploys a backbone model since the weights of 1Our code is available at https://github.com/ tangzhy/DPTDR the backbone model in each task are fine-tuned and therefore slightly different. That dramatically increases hardware costs and inefficiency. Recently, prompt tuning (PT) (Liu et al., 2021a) is a lightweight alternative to FT, which does not need storing a full copy of the backbone model for each task. One variant of PT, namely Deep Prompt Tuning (DPT; Li and Liang, 2021; Liu et al., 2021b), exhibits comparable performances with FT in various NLP tasks. DPT enjoys parameter- efficient(Houlsby et al., 2019) characteristics, of which the resulting prompts are light-weighted and can be easily passed to an online PLM service, thus overcoming the above challenge of FT. This paper asks: whether can we replace FT by DPT with com- parable performance to SOTA FT methods in dense passage retrieval? With comparable performance, DPT is much more friendly in deployment than FT. DPT usually freezes weights in the backbone models and alternatively trains deep prompts in- serted; the latter has much fewer parameters than the former. However, freezing most weights in DPT hinders its adaptability and therefore possi- bly harms performance. …

Content truncated to 500 words