Large Dual Encoders Are Generalizable Retrievers

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9844 - 9855 December 7-11, 2022 ©2022 Association for Computational Linguistics Large Dual Encoders Are Generalizable Retrievers Jianmo Ni∗, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernández Ábrego, Vincent Y. Zhao, Yi Luan, Keith B. Hall, Ming-Wei Chang, Yinfei Yang♣† Google Research, ♣Apple Abstract It has been shown that dual encoders trained on one domain often fail to generalize to other domains for retrieval tasks. One widespread belief is that the bottleneck layer of a dual en- coder, where the final score is simply a dot- product between a query vector and a pas- sage vector, is too limited compared to mod- els with fine-grained interactions between the query and the passage. In this paper, we chal- lenge this belief by scaling up the size of the dual encoder model while keeping the bottle- neck layer as a single dot-product with a fixed size. With multi-stage training, scaling up the model size brings significant improvement on a variety of retrieval tasks, especially for out- of-domain generalization. We further analyze the impact of the bottleneck layer and demon- strate diminishing improvement when scaling up the embedding size. Experimental results show that our dual encoders, Generalizable T5-based dense Retrievers (GTR), outperform previous sparse and dense retrievers on the BEIR dataset (Thakur et al., 2021) signifi- cantly. Most surprisingly, our ablation study finds that GTR is very data efficient, as it only needs 10% of MS Marco supervised data to match the out-of-domain performance of using all supervised data. 1 Introduction Typical neural retrieval models follow a dual en- coder paradigm (Gillick et al., 2018; Yang et al., 2020; Karpukhin et al., 2020). In this setup, queries and documents are encoded separately into a shared fixed-dimensional embedding space, where rele- vant queries and documents are represented in each other’s proximity. Then, approximated nearest neighbor search (Vanderkam et al., 2013; John- son et al., 2021) is applied to efficiently retrieve relevant documents given an encoded input query. ∗ Correspondence to jianmon@google.com † Work done while at Google Research Figure 1: Average Recall@100 and NDCG@10 on all BEIR tasks (excl. MS Marco). Scaling up consistently improves dual encoders’ out-of-domain performance. While dual encoders are popular neural retriev- ers, the expressiveness of the model is limited by a bottleneck layer consisting of only a simple dot- product between query and passage embeddings. Lu et al. (2021); Khattab and Zaharia (2020) argued that the dot-product (or cosine similarity) between the embeddings might not be powerful enough to capture the semantic relevance. Similarly, Thakur et al. (2021) suggested that dual encoder models have “issues for out-of-distribution data” and mod- els with more interactions between queries and documents have better generalization ability. In this paper, we challenge this belief by scaling up the dual encoder model size while keeping the bottleneck as a single dot-product with a fixed size. Note that scaling up a dual encoder is different from scaling up pretrained language models, such as …

Content truncated to 500 words