Incorporating Relevance Feedback for Information-Seeking Retrieval using Few-Shot Document Re-Ranking

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8988 - 9005 December 7-11, 2022 ©2022 Association for Computational Linguistics Incorporating Relevance Feedback for Information-Seeking Retrieval using Few-Shot Document Re-Ranking Tim Baumgärtner,1 Leonardo F. R. Ribeiro,1∗ Nils Reimers,2 Iryna Gurevych1 1Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science and Hessian Center for AI (hessian.AI), Technical University of Darmstadt 2cohere.ai www.ukp.tu-darmstadt.de Abstract Pairing a lexical retriever with a neural re- ranking model has set state-of-the-art perfor- mance on large-scale information retrieval datasets. This pipeline covers scenarios like question answering or navigational queries, however, for information-seeking scenarios, users often provide information on whether a document is relevant to their query in form of clicks or explicit feedback. Therefore, in this work, we explore how relevance feedback can be directly integrated into neural re-ranking models by adopting few-shot and parameter- efficient learning techniques. Specifically, we introduce a kNN approach that re-ranks docu- ments based on their similarity with the query and the documents the user considers relevant. Further, we explore Cross-Encoder models that we pre-train using meta-learning and subse- quently fine-tune for each query, training only on the feedback documents. To evaluate our dif- ferent integration strategies, we transform four existing information retrieval datasets into the relevance feedback scenario. Extensive exper- iments demonstrate that integrating relevance feedback directly in neural re-ranking models improves their performance, and fusing lexi- cal ranking with our best performing neural re- ranker outperforms all other methods by 5.2% nDCG@20.1 1 Introduction User queries can be categorized as navigational (retrieving a specific document), transactional (re- trieving a website to perform a particular action) or informational (Broder, 2002). For information- seeking queries, users might want to learn about a new topic or might be unfamiliar with the search domain. Therefore they potentially do not use com- mon keywords of the domain which decreases per- formance (Furnas et al., 1987). Furthermore, they *Now affiliated with Amazon Alexa AI. 1The code is available at https://github.com/UKPLab/ incorporating-relevance might want to find complementary information from diverse sources or consider different aspects of a topic (Clarke et al., 2008). Lastly, information- seeking queries can also be used to keep up with the latest developments on a topic. Concretely, these queries are encountered during scientific literature review (Voorhees et al., 2021; Dasigi et al., 2021), when looking for news and background information (Soboroff et al., 2018), during argument retrieval, (Bondarenko et al., 2021) or in the legal context for case law retrieval (Locke and Zuccon, 2018). Formulating effective queries to satisfy the com- plex information need in these scenarios is difficult. On the contrary, a user can easily judge whether a document is relevant to their query. Therefore, information obtained from the user when interact- ing with the search results, known as relevance feedback, can be used in the search. This can be ob- tained implicitly from click logs (Joachims, 2002) or explicitly by asking users whether a document is relevant (Rocchio, 1971). We focus on explicit feedback because it …

Content truncated to 500 words