Few-shot Reranking for Multi-hop QA via Language Model Prompting

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 15882–15897 July 9-14, 2023 ©2023 Association for Computational Linguistics Few-shot Reranking for Multi-hop QA via Language Model Prompting Muhammad Khalifa∗, Lajanugen Logeswaran†, Moontae Lee†‡, Honglak Lee∗†, Lu Wang∗ University of Michigan∗, LG AI Research†, University of Illinois at Chicago‡ Abstract We study few-shot reranking for multi-hop QA (MQA) with open-domain questions. To alle- viate the need for a large number of labeled question-document pairs for retriever training, we propose PROMPTRANK, which relies on language model prompting for multi-hop path reranking. PROMPTRANK first constructs an instruction-based prompt that includes a candi- date document path and then computes the rel- evance score between a given question and the path based on the conditional likelihood of the question given the path prompt according to a language model. PROMPTRANK yields strong retrieval performance on HotpotQA with only 128 training examples compared to state-of-the- art methods trained on thousands of examples — 73.6 recall@10 by PROMPTRANK vs. 77.8 by PathRetriever (Asai et al., 2020) and 77.5 by multi-hop dense retrieval (Xiong et al., 2021).1 1 Introduction Many information-seeking queries are in the form of multi-hop questions. For instance, to answer the question “What 1988 Christmas comedy film did Brian-Doyle Murray star in?”, we need to (i) search for movies starring Brian Murray, then (ii) identify which of them were released in 1988 dur- ing Christmas. Evidence required to answer such questions is often dispersed in different documents, requiring sequential, multi-step reasoning to reach the answer (Perez et al., 2020), typically referred to as multi-hop question answering (MQA). Given a multi-hop question and a large docu- ment corpus, existing MQA systems largely fol- low a retrieve-then-read pipeline, where a retriever module first identifies relevant documents from the corpus, and a reader module produces the an- swer based on the retrieved output (Asai et al., ∗Correspondence to khalifam@umich.edu 1Code available at https://github.com/mukhal/ PromptRank. 0 20 40 60 80 R@2 R@10 R@20 TF-IDF + BM25 DrKit PromptRank MDR Figure 1: Multi-hop retrieval recall@k on HotpotQA. PROMPTRANK, using only 128 examples, outperforms DrKit (Dhingra et al., 2020) and performs closely to Multi-hop Dense Retrieval (Xiong et al., 2021). Both fully supervised models are trained on ~90K examples. 2020; Li et al., 2021; Singh et al., 2021; Qi et al., 2021). The retriever module is trained to predict the ground-truth evidence document(s) given the ques- tion (Karpukhin et al., 2020; Qi et al., 2021; ). How- ever, curating large datasets of question-document pairs is expensive, especially for low-resource lan- guages or domains that require unique expertise (e.g., medical or legal documents), thus creating a bottleneck for building QA pipelines (Ram et al., 2022). Moreover, resorting to heuristics for data la- beling can lead to incorrect annotation (Izacard and Grave, 2021). This difficulty is further exacerbated in the case of multi-hop questions, as they need to be annotated with multiple support documents. The majority of existing data-efficient retrieval and reranking methods are restricted to …

Content truncated to 500 words