Boosted Dense Retriever

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3102 - 3117 July 10-15, 2022 ©2022 Association for Computational Linguistics Boosted Dense Retriever Patrick Lewis∗, Barlas O˘guz∗, Wenhan Xiong, Fabio Petroni, Wen-tau Yih, Sebastian Riedel Meta AI {plewis,barlaso,xwhan,fabiopetroni,scottyih,sriedel}@fb.com Abstract We propose DrBoost, a dense retrieval ensem- ble inspired by boosting. DrBoost is trained in stages: each component model is learned sequentially and specialized by focusing only on retrieval mistakes made by the current en- semble. The final representation is the concate- nation of the output vectors of all the compo- nent models, making it a drop-in replacement for standard dense retrievers at test time. Dr- Boost enjoys several advantages compared to standard dense retrieval models. It produces representations which are 4x more compact, while delivering comparable retrieval results. It also performs surprisingly well under approxi- mate search with coarse quantization, reducing latency and bandwidth needs by another 4x. In practice, this can make the difference be- tween serving indices from disk versus from memory, paving the way for much cheaper de- ployments.1 1 Introduction Identifying a small number of relevant documents from a large corpus to a given query, information retrieval is not only an important task in-and-of itself, but also plays a vital role in supporting a variety of knowledge-intensive NLP tasks (Lewis et al., 2020; Petroni et al., 2021), such as open- domain Question Answering (ODQA, Voorhees and Tice, 2000; Chen et al., 2017) and Fact Check- ing (Thorne et al., 2018). While traditional retrieval methods, such as TF-IDF and BM25 (Robertson, 2008), are built on sparse representations of queries and documents, dense retrieval approaches have shown superior performance recently on a range of retrieval and ranking tasks (Guu et al., 2020; Karpukhin et al., 2020; Reimers and Gurevych, 2019; Hofst¨atter et al., 2021b). Dense retrieval ∗Equal contribution 1Our code is available at https://github.com/ facebookresearch/drboost. involves embedding queries and documents as low- dimensional, continuous vectors, such that query and document embeddings are similar when the document is relevant to the query. The embedding function leverages the representational power of pretrained language models and is further finetuned using any available training query-document pairs. Document representations are computed offline in an index allowing dense retrieval to scale to mil- lions of documents, with query embeddings being computed on the fly. When deploying dense retrievers in real-world settings, however, there are two practical concerns: the size of the index and the retrieval time latency. The index size is largely determined by the num- ber of documents in the collection, as well as the embedding dimension. Whilst we cannot gener- ally control the former, reducing the embedding size is an attractive way to reduce index size. On lowering latency, Approximate Nearest-Neighbor (ANN) or Maximum Inner Product Search (MIPS) techniques are required in practice. This implies that it is far more important for retrieval models to perform well under approximate search rather than in the exact search setting. Developing a dense retrieval …

Content truncated to 500 words