Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies

Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies Mor Geva1,2, Daniel Khashabi2, Elad Segal1, Tushar Khot2, Dan Roth3, Jonathan Berant1,2 1Tel Aviv University 2Allen Institute for AI 3University of Pennsylvania morgeva@mail.tau.ac.il, {danielk,tushark}@allenai.org, elad.segal@gmail.com, danroth@seas.upenn.edu joberant@cs.tau.ac.il Abstract A key limitation in current datasets for multi- hop reasoning is that the required steps for answering the question are mentioned in it explicitly. In this work, we introduce STRATEGYQA, a question answering (QA) benchmark where the required reasoning steps are implicit in the question, and should be inferred using a strategy. A fundamental challenge in this setup how to elicit such creative questions fromis crowdsourcing workers, while covering a broad range of potential strategies. We propose a data collection procedure that combines term-based priming to inspire annotators, careful control over the annotator population, and adversarial filtering for eliminating reasoning shortcuts. Moreover, we annotate each question with (1) a decomposition into reasoning steps for answering it, and (2) Wikipedia paragraphs that contain the answers to each step. Overall, STRATEGYQA includes 2,780 examples, each consisting of a strategy question, its decomposition, and evidence paragraphs. Analysis shows that questions in STRATEGYQA are short, topic-diverse, and cover a wide range of strategies. Empirically, we show that humans perform well (87%) on this task, while our best baseline reaches an accuracy of ∼ 66%. 1 Introduction Developing models that successfully reason over multiple parts of their input has attracted substantial attention recently, leading to the creation of many multi-step reasoning Question Answering (QA) benchmarks (Welbl et al., 2018; Talmor and Berant, 2018; Khashabi et al., 2018; Yang et al., 2018; Dua et al., 2019; Suhr et al., 2019). Commonly, the language of questions in such benchmarks explicitly describes the process for deriving the answer. For instance (Figure 1, Q2), the question Was Aristotle alive when the laptop was invented? explicitly specifies the required reasoning steps. However, in real-life questions, reasoning is often implicit. For example, the question Did Aristotle use a laptop? (Q1) can be answered using the same steps, but the model must infer the strategy for answering the question–temporal comparison, in this case. Answering implicit questions poses several challenges compared to answering their explicit counterparts. First, retrieving the context is difficult as there is little overlap between the question and its context (Figure 1, Q1 and ‘E’). Moreover, questions tend to be short, lowering the possibility of the model exploiting shortcuts in the language of the question. In this work, we introduce STRATEGYQA, a Boolean QA benchmark focusing on implicit multi-hop reasoning for strategy questions, where a strategy is the ability to infer from a question its atomic sub- questions. In contrast to previous benchmarks (Khot et al., 2020a; Yang et al., 2018), questions in STRATEGYQA are not limited to predefined decomposition patterns and cover a wide range of strategies that humans apply when answering questions. Eliciting strategy questions using crowdsourc- ing is non-trivial. First, authoring such questions requires creativity. Past work often collected multi-hop questions by showing workers an entire …

Content truncated to 500 words