Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 10014–10037 July 9-14, 2023 ©2023 Association for Computational Linguistics Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions Harsh Trivedi† Niranjan Balasubramanian† †Stony Brook University Stony Brook, U.S.A. {hjtrivedi,niranjan}@cs.stonybrook.edu Tushar Khot‡ Ashish Sabharwal‡ ‡Allen Institute for AI Seattle, U.S.A. {tushark,ashishs}@allenai.org Abstract Prompting-based large language models (LLMs) are surprisingly powerful at gener- ating natural language reasoning steps or Chains-of-Thoughts (CoT) for multi-step question answering (QA). They struggle, however, when the necessary knowledge is either unavailable to the LLM or not up-to-date within its parameters. While using the question to retrieve relevant text from an external knowledge source helps LLMs, we observe that this one-step retrieve-and-read approach is insufficient for multi-step QA. Here, what to retrieve depends on what has already been derived, which in turn may depend on what was previously retrieved. To address this, we propose IRCoT, a new approach for multi-step QA that interleaves retrieval with steps (sentences) in a CoT, guiding the retrieval with CoT and in turn using retrieved results to improve CoT. Using IRCoT with GPT3 substantially improves retrieval (up to 21 points) as well as downstream QA (up to 15 points) on four datasets: HotpotQA, 2WikiMultihopQA, MuSiQue, and IIRC. We observe similar substantial gains in out-of- distribution (OOD) settings as well as with much smaller models such as Flan-T5-large without additional training. IRCoT reduces model hallucination, resulting in factually more accurate CoT reasoning.1. 1 Introduction Large language models are capable of answer- ing complex questions by generating step-by- step natural language reasoning steps—so called chains of thoughts (CoT)—when prompted appro- priately (Wei et al., 2022). This approach has been successful when all information needed to answer the question is either provided as context (e.g., al- gebra questions) or assumed to be present in the model’s parameters (e.g., commonsense reasoning). 1Code, data, and prompts are available at https:// github.com/stonybrooknlp/ircot In what country was Lost Gravity manufactured? The Lost Gravity was manufactured by Mack Rides. Mack Rides is a company from Germany. The answer is Germany. cumulate docs cumulate docs cumulate docs Figure 1: IRCoT interleaves chain-of-thought (CoT) generation and knowledge retrieval steps in order to guide the retrieval by CoT and vice-versa. This inter- leaving allows retrieving more relevant information for later reasoning steps, compared to standard retrieval us- ing solely the question as the query. However, for many open-domain questions, all re- quired knowledge is not always available or up-to- date in models’ parameters and it’s beneficial to retrieve knowledge from external sources (Lazari- dou et al., 2022; Kasai et al., 2022). How can we augment chain-of-thought prompt- ing for open-domain, knowledge-intensive tasks that require complex, multi-step reasoning? While a one-shot retrieval from a knowledge source based solely on the question can success- fully augment LMs with relevant knowledge for many factoid-based tasks (Lewis et al., 2020; Guu et al., 2020; Borgeaud et al., 2022; Izacard et al., 2022), this strategy has clear limitations for more complex multi-step …

Content truncated to 500 words