Decoding a Neural Retriever’s Latent Space for Query Suggestion

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8786 - 8804 December 7-11, 2022 ©2022 Association for Computational Linguistics Decoding a Neural Retriever’s Latent Space for Query Suggestion Leonard Adolphs† Michelle Chen Huebscher‡ Christian Buck‡ Sertan Girgin‡ Olivier Bachem‡ Massimiliano Ciaramita‡ Thomas Hofmann† †ETH Zürich ladolphs@inf.ethz.ch ‡Google Research Abstract Neural retrieval models have superseded clas- sic bag-of-words methods such as BM25 as the retrieval framework of choice. However, neural systems lack the interpretability of bag- of-words models; it is not trivial to connect a query change to a change in the latent space that ultimately determines the retrieval results. To shed light on this embedding space, we learn a “query decoder” that, given a latent repre- sentation of a neural search engine, generates the corresponding query. We show that it is possible to decode a meaningful query from its latent representation and, when moving in the right direction in latent space, to decode a query that retrieves the relevant paragraph. In particular, the query decoder can be useful to understand “what should have been asked” to retrieve a particular paragraph from the col- lection. We employ the query decoder to gen- erate a large synthetic dataset of query refor- mulations for MSMarco, leading to improved retrieval performance. On this data, we train a pseudo-relevance feedback (PRF) T5 model for the application of query suggestion that out- performs both query reformulation and PRF information retrieval baselines. 1 Introduction Neural encoder models (Karpukhin et al., 2020; Ni et al., 2021; Izacard et al., 2021) have improved document retrieval in various settings. They have become an essential building block for applications in open-domain question answering (Karpukhin et al., 2020; Lewis et al., 2020b; Izacard and Grave, 2021), open-domain conversational agents (Shuster et al., 2021; Adolphs et al., 2021), and, recently, language modeling (Shuster et al., 2022). Neural encoders embed documents and queries in a shared (or joint) latent space, so that paragraphs can be ranked and retrieved based on their vector similar- ity with a given query. This constitutes a concep- tually powerful approach to discovering semantic similarities between queries and documents that is often found to be more nuanced than simple term frequency statistics typical of classic sparse rep- resentations. However, such encoders may come with shortcomings in practice. First, they are prone to domain overfitting, failing to consistently outper- form bag-of-words approaches on out-of-domain queries (Thakur et al., 2021). Second, they are no- toriously hard to interpret as similarity is no longer controlled by word overlap, but rather by seman- tic similarities that lack explainability. Third, they may be non-robust as small changes in the query can lead to inexplicably different retrieval results. In bag-of-words models, it can be straightfor- ward to modify a query to retrieve a given docu- ment: e.g., following insights from relevance feed- back (Rocchio, 1971), by increasing the weight of terms contained in the target document (Adolphs et al., 2022; Huebscher et al., 2022). This approach is not trivially applicable to …

Content truncated to 500 words