RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5835–5847 June 6–11, 2021. ©2021 Association for Computational Linguistics 5835 RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering Yingqi Qu1, Yuchen Ding1, Jing Liu1∗, Kai Liu1, Ruiyang Ren2† Wayne Xin Zhao2∗, Daxiang Dong1, Hua Wu1 and Haifeng Wang1 1Baidu Inc.; 2Gaoling School of Artificial Intelligence, Renmin University of China {quyingqi, dingyuchen, liujing46, liukai20, dongdaxiang, wu_hua, wanghaifeng}@baidu.com reyon.ren@ruc.edu.cn, batmanfly@gmail.com Abstract In open-domain question answering, dense passage retrieval has become a new paradigm to retrieve relevant passages for finding an- swers. Typically, the dual-encoder architec- ture is adopted to learn dense representations of questions and passages for semantic match- ing. However, it is difficult to effectively train a dual-encoder due to the challenges includ- ing the discrepancy between training and infer- ence, the existence of unlabeled positives and limited training data. To address these chal- lenges, we propose an optimized training ap- proach, called RocketQA, to improving dense passage retrieval. We make three major techni- cal contributions in RocketQA, namely cross- batch negatives, denoised hard negatives and data augmentation. The experiment results show that RocketQA significantly outperforms previous state-of-the-art models on both MS- MARCO and Natural Questions. We also con- duct extensive experiments to examine the ef- fectiveness of the three strategies in RocketQA. Besides, we demonstrate that the performance of end-to-end QA can be improved based on our RocketQA retriever 1. 1 Introduction Open-domain question answering (QA) aims to find the answers to natural language questions from a large collection of documents. Early QA systems (Brill et al., 2002; Dang et al., 2007; Ferrucci et al., 2010) constructed complicated pipelines consist- ing of multiple components, including question understanding, document retrieval, passage rank- ing and answer extraction. Recently, inspired by the advancements of machine reading comprehen- sion (MRC), Chen et al. (2017) proposed a sim- plified two-stage approach, where a traditional IR ∗ Corresponding authors. †† The work was done when Ruiyang Ren was doing in- ternship at Baidu. 1Our code is available at https://github.com/ PaddlePaddle/Research/tree/master/NLP/ NAACL2021-RocketQA [CLS] q(1) q(k) [CLS] ... p(1) p(l) ... ... ... ... ... Eq(q) Ep(p) sim(q,�p)�=�Eq(q)�Ep(p) question passage (a) A dual-encoder based on pre-trained LMs. ... ... ... ... [CLS] q(1) q(k) [SEP] ... p(1) p(l) ... sim(q, p) question passage (b) A cross-encoder based on pre-trained LMs. Figure 1: The comparison of dual-encoder and cross- encoder architectures. retriever (e.g., TF-IDF or BM25) first selects a few relevant passages as contexts, and then a neu- ral reader reads the contexts and extracts the an- swers. As the recall component, the first-stage retriever significantly affects the final QA perfor- mance. Though efficient with an inverted index, traditional IR retrievers with term-based sparse rep- resentations have limited capabilities in matching questions and passages, e.g., term mismatch. To deal with the issue of term mismatch, the dual-encoder architecture (as shown in Figure 1a) has been widely explored (Lee et al., 2019; Guu et al., …

Content truncated to 500 words