Compositional Questions Do Not Necessitate Multi-hop Reasoning

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4249–4257 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 4249 Compositional Questions Do Not Necessitate Multi-hop Reasoning Sewon Min∗ 1, Eric Wallace∗ 2, Sameer Singh3, Matt Gardner2, Hannaneh Hajishirzi1,2, Luke Zettlemoyer1 1University of Washington 2Allen Institute for Artificial Intelligence 3University of California, Irvine sewon@cs.washington.edu, ericw@allenai.org Abstract Multi-hop reading comprehension (RC) ques- tions are challenging because they require reading and reasoning over multiple para- graphs. We argue that it can be difficult to con- struct large multi-hop RC datasets. For exam- ple, even highly compositional questions can be answered with a single hop if they target specific entity types, or the facts needed to answer them are redundant. Our analysis is centered on HOTPOTQA, where we show that single-hop reasoning can solve much more of the dataset than previously thought. We intro- duce a single-hop BERT-based RC model that achieves 67 F1—comparable to state-of-the- art multi-hop models. We also design an eval- uation setting where humans are not shown all of the necessary paragraphs for the intended multi-hop reasoning but can still answer over 80% of questions. Together with detailed error analysis, these results suggest there should be an increasing focus on the role of evidence in multi-hop reasoning and possibly even a shift towards information retrieval style evaluations with large and diverse evidence collections. 1 Introduction Multi-hop reading comprehension (RC) requires reading and aggregating information over multi- ple pieces of textual evidence (Welbl et al., 2017; Yang et al., 2018; Talmor and Berant, 2018). In this work, we argue that it can be difficult to con- struct large multi-hop RC datasets. This is because multi-hop reasoning is a characteristic of both the question and the provided evidence; even highly compositional questions can be answered with a single hop if they target specific entity types, or the facts needed to answer them are redundant. For example, the question in Figure 1 is compositional: a plausible solution is to find “What animal’s habi- tat was the R´eserve Naturelle Lomako Yokokala ∗Equal Contribution. Question: What is the former name of the animal whose habitat the R´eserve Naturelle Lomako Yokokala was es- tablished to protect? Paragraph 5: The Lomako Forest Reserve is found in Democratic Republic of the Congo. It was established in 1991 especially to protect the habitat of the Bonobo apes. Paragraph 1: The bonobo (“Pan paniscus”), formerly called the pygmy chimpanzee and less often, the dwarf or gracile chimpanzee, is an endangered great ape and one of the two species making up the genus “Pan”. Figure 1: A HOTPOTQA example designed to require reasoning across two paragraphs. Eight spurious addi- tional paragraphs (not shown) are provided to increase the task difficulty. However, since only one of the ten paragraphs is about an animal, one can immediately lo- cate the answer in Paragraph 1 using one hop. The full example is provided in Appendix A. established to protect?”, and then answer “What is the …

Content truncated to 500 words