DuReader-Retrieval: A Large-scale Chinese Benchmark for Passage Retrieval from Web Search Engine

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5326 - 5338 December 7-11, 2022 ©2022 Association for Computational Linguistics DuReaderretrieval: A Large-scale Chinese Benchmark for Passage Retrieval from Web Search Engine Yifu Qiu1†, Hongyu Li2, Yingqi Qu2, Ying Chen2, Qiaoqiao She2, Jing Liu2*, Hua Wu2, Haifeng Wang2 1Institute for Language, Cognition and Computation, University of Edinburgh, UK 2Baidu Inc., Beijing, China y.qiu-20@sms.ed.ac.uk {lihongyu04, quyingqi, chenying04, sheqiaoqiao, liujing46, wu_hua, wanghaifeng}@baidu.com Abstract In this paper, we present DuReaderretrieval, a large-scale Chinese dataset for passage re- trieval. DuReaderretrieval contains more than 90K queries and over 8M unique passages from a commercial search engine. To alle- viate the shortcomings of other datasets and ensure the quality of our benchmark, we (1) reduce the false negatives in development and test sets by manually annotating results pooled from multiple retrievers, and (2) re- move the training queries that are semanti- cally similar to the development and testing queries. Additionally, we provide two out- of-domain testing sets for cross-domain eval- uation, as well as a set of human translated queries for for cross-lingual retrieval eval- uation. The experiments demonstrate that DuReaderretrieval is challenging and a num- ber of problems remain unsolved, such as the salient phrase mismatch and the syntactic mis- match between queries and paragraphs. These experiments also show that dense retrievers do not generalize well across domains, and cross-lingual retrieval is essentially challeng- ating. DuReaderretrieval is publicly available https://github.com/baidu/DuReader/ tree/master/DuReader-Retrieval. 1 Introduction Passage retrieval requires systems to select can- didate passages from a large passage collection. In recent years, pre-trained language models (De- vlin et al., 2019; Liu et al., 2019) have been ap- plied to retrieval problems, known as dense re- trieval (Karpukhin et al., 2020; Qu et al., 2021; Zhan et al., 2021). The success of dense retrieval relies on the availability of high quality, large-scale, human-annotated corpora. A number of popular datasets are already available for English passage retrieval, including MS-MARCO (Nguyen et al., †The work was done when the first author was doing in- ternship at Baidu. *Corresponding author. 2016), TriviaQA (Joshi et al., 2017), and Natural Questions (Kwiatkowski et al., 2019). In contrast, existing datasets for non-English retrieval (e.g., Chinese), are either small or machine generated. For example, TianGong-PDR (Wu et al., 2019) has only 70 questions and 11K passages. Even though the multilingual dataset mMARCO(Bonifacio et al., 2021) is large in size, it is constructed by ma- chine translation from the English MS-MARCO dataset. Sougou-QCL (Zheng et al., 2018) is con- structed based on click logs of web data with- out human annotation. In this paper, we present DuReaderretrieval, a large-scale Chinese dataset for passage retrieval from web search engine, that is manually annotated. The dataset contains more than 90K queries and over 8M unique passages. All queries are selected from real requests made by users at Baidu Search, and document passages are from the search results. Similar to (Karpukhin et al., 2020), we create the DuReaderretrieval from DuReader (He et al., 2018), a …

Content truncated to 500 words