PAIR: Leveraging Passage-Centric Similarity Relation for Improving Dense Passage Retrieval

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2173–2183 August 1–6, 2021. ©2021 Association for Computational Linguistics 2173 PAIR: Leveraging Passage-Centric Similarity Relation for Improving Dense Passage Retrieval Ruiyang Ren1,3∗†, Shangwen Lv2∗, Yingqi Qu2, Jing Liu2‡, Wayne Xin Zhao3,4‡ Qiaoqiao She2, Hua Wu2, Haifeng Wang2 and Ji-Rong Wen3,4 1School of Information, Renmin University of China; 2Baidu Inc. 3Beijing Key Laboratory of Big Data Management and Analysis Methods 4Gaoling School of Artificial Intelligence, Renmin University of China {reyon.ren, jrwen}@ruc.edu.cn, batmanfly@gmail.com {lvshangwen, quyingqi, liujing46, sheqiaoqiao, wu_hua, wanghaifeng}@baidu.com Abstract Recently, dense passage retrieval has become a mainstream approach to finding relevant infor- mation in various natural language processing tasks. A number of studies have been devoted to improving the widely adopted dual-encoder architecture. However, most of the previous studies only consider query-centric similarity relation when learning the dual-encoder re- triever. In order to capture more comprehen- sive similarity relations, we propose a novel approach that leverages both query-centric and PAssage-centric sImilarity Relations (called PAIR) for dense passage retrieval. To im- plement our approach, we make three major technical contributions by introducing formal formulations of the two kinds of similarity relations, generating high-quality pseudo la- beled data via knowledge distillation, and de- signing an effective two-stage training proce- dure that incorporates passage-centric similar- ity relation constraint. Extensive experiments show that our approach significantly outper- forms previous state-of-the-art models on both MSMARCO and Natural Questions datasets1. 1 Introduction With the recent advances of pre-trained language models, dense passage retrieval techniques (repre- senting queries and passages in low-dimensional semantic space) have significantly outperformed traditional term-based techniques (Guu et al., 2020; Karpukhin et al., 2020). As the key step of finding the relevant information, it has been shown that dense passage retrieval can effectively im- prove the performance in a variety of tasks, includ- ∗ Equal contribution. † The work was done when Ruiyang Ren was doing in- ternship at Baidu. ‡ Corresponding authors. 1Our code is available at https://github.com/ PaddlePaddle/Research/tree/master/NLP/ ACL2021-PAIR q p- p+ (a) q p- p+ (b) Figure 1: An illustrative case of a query q, its positive passage p+ and negative passage p−: (a) Query-centric similarity relation enforces s(q, p+) > s(q, p−); (b) Passage-centric similarity relation further enforces s(p+, q) > s(p+, p−), where s(p+, q) = s(q, p+). We use the distance (i.e., dissimilarity) for visualization: the longer the distance is, the less similar it is. ing question answering (Lee et al., 2019; Xiong et al., 2020b), information retrieval (Luan et al., 2021; Khattab and Zaharia, 2020), dialogue (Ji et al., 2014; Henderson et al., 2017) and entity linking (Gillick et al., 2019; Wu et al., 2020). Typically, the dual-encoder architecture is used to learn the dense representations of queries and passages, and the dot-product similarity between the representations of queries and passages be- comes ranking measurement for retrieval. A num- ber of studies have been devoted to improving this architecture (Guu et al., 2020; Karpukhin et al., 2020; Xiong et al., 2020a) for dense passage re- trieval. Previous …

Content truncated to 500 words