What Are You Token About? Dense Retrieval as Distributions Over the Vocabulary

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 2481–2498 July 9-14, 2023 ©2023 Association for Computational Linguistics What Are You Token About? Dense Retrieval as Distributions Over the Vocabulary Ori Ram1 Liat Bezalel1 Adi Zicher1 Yonatan Belinkov2∗ Jonathan Berant1 Amir Globerson1 1Blavatnik School of Computer Science, Tel Aviv University 2Technion – IIT, Israel ori.ram@cs.tau.ac.il, liatbezalel@mail.tau.ac.il, adizicher@mail.tau.ac.il belinkov@technion.ac.il, joberant@cs.tau.ac.il, gamir@tauex.tau.ac.il Abstract Dual encoders are now the dominant architec- ture for dense retrieval. Yet, we have little un- derstanding of how they represent text, and why this leads to good performance. In this work, we shed light on this question via distributions over the vocabulary. We propose to interpret the vector representations produced by dual encoders by projecting them into the model’s vocabulary space. We show that the resulting projections contain rich semantic information, and draw connection between them and sparse retrieval. We find that this view can offer an explanation for some of the failure cases of dense retrievers. For example, we observe that the inability of models to handle tail entities is correlated with a tendency of the token distri- butions to forget some of the tokens of those entities. We leverage this insight and propose a simple way to enrich query and passage repre- sentations with lexical information at inference time, and show that this significantly improves performance compared to the original model in zero-shot settings, and specifically on the BEIR benchmark.1 1 Introduction Dense retrieval models based on neural text repre- sentations have proven very effective (Karpukhin et al., 2020; Qu et al., 2021; Ram et al., 2022; Izacard et al., 2022a,b), improving upon strong tra- ditional sparse models like BM25 (Robertson and Zaragoza, 2009). However, when applied off-the- shelf (i.e., in out-of-domain settings) they often experience a severe drop in performance (Thakur et al., 2021; Sciavolino et al., 2021; Reddy et al., 2021). Moreover, the reasons for such failures are poorly understood, as the information captured in their representations remains under-investigated. ∗Supported by the Viterbi Fellowship in the Center for Computer Engineering at the Technion. 1Our code is publicly available at https://github. com/oriram/dense-retrieval-projections. Uninterpretable Q: Where was Michael Jack born? Query Encoder MLM Head michael -0.54 jack -1.39 son -4.19 father -4.58 birth -4.83 boy -5.15 family -5.75 locke -5.78 childhood -6.10 baby -6.11 child -6.15 ⋮ Interpretable CLS where was michael jack born ? Figure 1: An example of our framework. We run the question “Where was Michael Jack born?” through the question encoder of DPR (Karpukhin et al., 2020), and project the question representation eq to the vocabulary space using BERT’s masked language modeling head (Devlin et al., 2019). The result is a distribution over the vocabulary, Q. We apply the same procedure for passages as well. These projections enable reasoning about and improving retrieval representations. In this work, we present a new approach for interpreting and reasoning about dense retrievers, through distributions induced by their query2 and passage representations when projected to the vo- cabulary …

Content truncated to 500 words