TOME: A Two-stage Approach for Model-based Retrieval

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 6102–6114 July 9-14, 2023 ©2023 Association for Computational Linguistics TOME: A Two-stage Approach for Model-based Retrieval Ruiyang Ren1,3∗ Wayne Xin Zhao1,3† Jing Liu2† Hua Wu2 Ji-Rong Wen1,3 Haifeng Wang2 1Gaoling School of Artificial Intelligence, Renmin University of China 2Baidu Inc. 3Beijing Key Laboratory of Big Data Management and Analysis Methods {reyon.ren, jrwen}@ruc.edu.cn, batmanfly@gmail.com {liujing46, wu_hua, wanghaifeng}@baidu.com Abstract Recently, model-based retrieval has emerged as a new paradigm in text retrieval that discards the index in the traditional retrieval model and instead memorizes the candidate corpora us- ing model parameters. This design employs a sequence-to-sequence paradigm to generate document identifiers, which enables the com- plete capture of the relevance between queries and documents and simplifies the classic index- retrieval-rerank pipeline. Despite its attractive qualities, there remain several major challenges in model-based retrieval, including the discrep- ancy between pre-training and fine-tuning, and the discrepancy between training and inference. To deal with the above challenges, we propose a novel two-stage model-based retrieval ap- proach called TOME, which makes two major technical contributions, including the utiliza- tion of tokenized URLs as identifiers and the design of a two-stage generation architecture. We also propose a number of training strategies to deal with the training difficulty as the corpus size increases. Extensive experiments and anal- ysis on MS MARCO and Natural Questions demonstrate the effectiveness of our proposed approach, and we investigate the scaling laws of TOME by examining various influencing factors. 1 Introduction Information retrieval systems have undergone con- tinuous development over the past few decades, with the aim of obtaining relevant resources, such as documents, in response to a user query from a vast collection. With the recent success of Pre- trained Language Models (PLMs) (Devlin et al., 2019; Raffel et al., 2020; Zhao et al., 2023), re- searchers have developed PLM-based dense retriev- ers (Lin et al., 2021; Zhao et al., 2022), which utilize dual-encoders and nearest neighbor search ∗The work was done during the internship at Baidu. † Corresponding authors. index for retrieval and achieve significant improve- ments over sparse retrievers. More recently, a new retrieval paradigm, known as model-based retrieval (Tay et al., 2022; Zhou et al., 2022c), has been introduced by developing an alternative architecture for retrieval. In contrast to traditional retrieval methods, it does not explic- itly maintain a corpus index, thereby simplifying the classic index-retrieve-rerank process. Typically, a model-based retrieval system is built based on a sequence-to-sequence generation model with an encoder-decoder architecture, such as T5 (Raffel et al., 2020) and BART (Lewis et al., 2020). It accepts a query as input and directly generates the corresponding document identifier via the genera- tion model. Despite its attractive benefits in simplifying the retrieval pipeline, model-based retrieval still faces following major challenges. • Firstly, since the retrieval task is framed as a prediction task of document identifiers, mak- ing it crucial to design document identifiers that are well-suited to the underlying generative PLM. However, …

Content truncated to 500 words