RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-ranking

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2825–2835 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 2825 RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-ranking Ruiyang Ren1,3∗, Yingqi Qu2∗, Jing Liu2, Wayne Xin Zhao3,4†, Qiaoqiao She2 Hua Wu2†, Haifeng Wang2 and Ji-Rong Wen1,3,4 1School of Information, Renmin University of China; 2Baidu Inc. 3Beijing Key Laboratory of Big Data Management and Analysis Methods 4Gaoling School of Artificial Intelligence, Renmin University of China {reyon.ren, jrwen}@ruc.edu.cn, batmanfly@gmail.com {quyingqi, liujing46, sheqiaoqiao, wu_hua, wanghaifeng}@baidu.com Abstract In various natural language processing tasks, passage retrieval and passage re-ranking are two key procedures in finding and ranking rel- evant information. Since both the two pro- cedures contribute to the final performance, it is important to jointly optimize them in order to achieve mutual improvement. In this pa- per, we propose a novel joint training approach for dense passage retrieval and passage re- ranking. A major contribution is that we intro- duce the dynamic listwise distillation, where we design a unified listwise training approach for both the retriever and the re-ranker. During the dynamic distillation, the retriever and the re-ranker can be adaptively improved accord- ing to each other’s relevance information. We also propose a hybrid data augmentation strat- egy to construct diverse training instances for listwise training approach. Extensive experi- ments show the effectiveness of our approach on both MSMARCO and Natural Questions datasets. Our code is available at https:// github.com/PaddlePaddle/RocketQA. 1 Introduction Recently, dense passage retrieval has become an important approach in the task of passage re- trieval (Karpukhin et al., 2020) to identify rele- vant contents from a large corpus. The underlying idea is to represent both queries and passages as low-dimensional vectors (a.k.a., embeddings), so that the relevance can be measured via embedding similarity. Additionally, a subsequent procedure of passage re-ranking is widely adopted to further improve the retrieval results by incorporating a re- ranker (Qu et al., 2021; Luan et al., 2021). Such a two-stage procedure is particularly useful in a va- riety of natural language processing tasks, includ- ing question answering (Mao et al., 2021; Xiong ∗ Equal contribution. The work was done when Ruiyang Ren was doing internship at Baidu. † Corresponding authors. et al., 2020b), dialogue system (Ji et al., 2014; Henderson et al., 2017) and entity linking (Gillick et al., 2019; Wu et al., 2020). Following a retrieve-then-rerank way, the dense retriever in passage retrieval and the re-ranker in passage re-ranking jointly contribute to the final performance. Despite the fact that the two mod- ules work as a pipeline during the inference stage, it has been found useful to train them in a corre- lated manner. For example, the retriever with a dual-encoder can be improved by distilling from the re-ranker with a more capable cross-encoder architecture (Qu et al., 2021; Yang et al., 2020), and the re-ranker can be improved with training instances generated from the retriever (Qu et al., 2021; Huang et al., 2020; Gao et al., 2021b). …

Content truncated to 500 words