Understanding Dataset Design Choices for Multi-hop Reasoning

Proceedings of NAACL-HLT 2019, pages 4026–4032 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 4026 Understanding Dataset Design Choices for Multi-hop Reasoning Jifan Chen and Greg Durrett The University of Texas at Austin {jfchen,gdurrett}@cs.utexas.edu Abstract Learning multi-hop reasoning has been a key challenge for reading comprehension models, leading to the design of datasets that explic- itly focus on it. Ideally, a model should not be able to perform well on a multi-hop ques- tion answering task without doing multi-hop reasoning. In this paper, we investigate two recently proposed datasets, WikiHop (Welbl et al., 2018) and HotpotQA (Yang et al., 2018). First, we explore sentence-factored models for these tasks; by design, these models cannot do multi-hop reasoning, but they are still able to solve a large number of examples in both datasets. Furthermore, we find spurious corre- lations in the unmasked version of WikiHop, which make it easy to achieve high perfor- mance considering only the questions and an- swers. Finally, we investigate one key dif- ference between these datasets, namely span- based vs. multiple-choice formulations of the QA task. Multiple-choice versions of both datasets can be easily gamed, and two models we examine only marginally exceed a baseline in this setting. Overall, while these datasets are useful testbeds, high-performing models may not be learning as much multi-hop reasoning as previously thought. 1 Introduction Question answering from text (Richardson et al., 2013; Hill et al., 2015; Hermann et al., 2015; Ra- jpurkar et al., 2016) is a key challenge problem for NLP that tests whether models can extract infor- mation based on a query. However, even sophis- ticated models that perform well on QA bench- marks (Seo et al., 2017; Shen et al., 2017; Yu et al., 2018) may only be doing shallow pattern match- ing of the question against the supporting passage (Weissenborn et al., 2017). More recent work (Ku- mar et al., 2016; Joshi et al., 2017; Welbl et al., 2018) has emphasized gathering information from different parts of a passage to answer the ques- tion, leading to a number of models designed to do multi-hop reasoning. Two recent large-scale datasets have been specifically designed to test multi-hop reasoning: WikiHop (Welbl et al., 2018) and HotpotQA (Yang et al., 2018). In this paper, we seek to answer two main ques- tions. First, although the two datasets are explic- itly constructed for multi-hop reasoning, do mod- els really need to do multi-hop reasoning to do well on them? Recent work has shown that large- scale QA datasets often do not exhibit their adver- tised properties (Chen et al., 2016; Kaushik and Lipton, 2018). We devise a test setting to see whether multi-hop reasoning is necessary: can a model which treats each sentence independently select the sentence containing the answer? This provides a rough estimate of the fraction of ques- tions solvable by a non-multi-hop system. Our results show that more than half of the questions in WikiHop and HotpotQA do not require multi- hop reasoning …

Content truncated to 500 words