♫ MuSiQue: Multihop Questions via Single-hop Question Composition

� MuSiQue: Multihop Questions via Single-hop Question Composition Harsh Trivedi⋄ Niranjan Balasubramanian⋄ Tushar Khot† Ashish Sabharwal† ⋄Stony Brook University, Stony Brook, USA {hjtrivedi,niranjan}@cs.stonybrook.edu †Allen Institute for AI, Seattle, USA {tushark,ashishs}@allenai.org Abstract Multihop reasoning remains an elusive goal as existing multihop benchmarks are known to be largely solvable via shortcuts. Can we create a question answering (QA) dataset that, by construction, requires proper multihop rea- soning? To this end, we introduce a bottom–up approach that systematically selects compos- able pairs of single-hop questions that are connected, that is, where one reasoning step critically relies on information from another. This bottom–up methodology lets us explore a vast space of questions and add stringent filters as well as other mechanisms targeting connected reasoning. It provides fine-grained control over the construction process and the properties of the resulting k-hop questions. We use this methodology to create MuSiQue-Ans, a new multihop QA dataset with 25K 2–4 hop questions. Relative to existing datasets, MuSiQue-Ans is more difficult overall (3× increase in human–machine gap), and harder to cheat via disconnected reasoning (e.g., a single-hop model has a 30-point drop in F1). We further add unanswerable contrast ques- tions to produce a more stringent dataset, MuSiQue-Full. We hope our datasets will help the NLP community develop models that perform genuine multihop reasoning.1 1 Introduction Multihop QA datasets are designed to support the development and evaluation of models that perform multiple steps of reasoning in order to answer a question. Recent work, however, shows that on existing datasets, models often need not even connect information across all supporting 1Code and datasets available at https://github .com/stonybrooknlp/musique. facts,2 because they can exploit reasoning short- cuts and other artifacts to find the correct answers and obtain high scores (Min et al., 2019a; Chen and Durrett, 2019; Trivedi et al., 2020). Such short- cuts arise from various factors, such as overly specific sub-questions, train-test leakage, and in- sufficient distractors. These factors allow models to circumvent connected reasoning—they need not read the context to find answers to previous sub-question(s) or use these answers to answer the later sub-questions that depend on them. The left hand side of Fig. 1 illustrates an in- stance of this problem in an actual question (Q) taken from the HotpotQA dataset (Yang et al., 2018). This question has the over-specification is- sue. At first glance, it appears to require a model to identify Kurt Vonnegut as the author of Armaged- don in Retrospect, and then use this information to answer the final question about the famous satire novel he authored. However, this framing of the question is insufficient to enforce that models must perform connected multihop reasoning to arrive at the correct answer. A model can, in fact, find the correct answer to this question from the context without finding the answer to Q1. This is because, even if a model does not know that A1 refers to Kurt Vonnegut, there happens to be only one person best known for a satirical novel mentioned in the context. Contrast this …

Content truncated to 500 words