Break It Down: A Question Understanding Benchmark

BREAK It Down: A Question Understanding Benchmark Tomer Wolfson1,3, Mor Geva1,3, Ankit Gupta1, Matt Gardner3, Yoav Goldberg2,3, Daniel Deutch1, Jonathan Berant1,3 1Tel Aviv University 2Bar-Ilan University 3Allen Institute for AI {tomerwol,morgeva}@mail.tau.ac.il, ankitgupta.iitkanpur@gmail.com, mattg@allenai.org, yoav.goldberg@gmail.com, danielde@post.tau.ac.il, joberant@cs.tau.ac.il Abstract Understanding natural language questions en- tails the ability to break down a question into the requisite steps for computing its answer. In this work, we introduce a Question Decom- position Meaning Representation (QDMR) for questions. QDMR constitutes the ordered list of steps, expressed through natural language, that are necessary for answering a question. We develop a crowdsourcing pipeline, show- ing that quality QDMRs can be annotated at scale, and release the BREAK dataset, con- taining over 83K pairs of questions and their QDMRs. We demonstrate the utility of QDMR by showing that (a) it can be used to im- prove open-domain question answering on the HOTPOTQA dataset, (b) it can be determin- istically converted to a pseudo-SQL formal language, which can alleviate annotation in semantic parsing applications. Last, we use BREAK to train a sequence-to-sequence model with copying that parses questions into QDMR structures, and show that it substantially out- performs several natural baselines. 1 Introduction Recently, increasing work has been devoted to models that can reason and integrate information from multiple parts of an input. This includes rea- soning over images (Antol et al., 2015; Johnson et al., 2017; Suhr et al., 2019;HudsonandManning, 2019), paragraphs (Dua et al., 2019), documents (Welbl et al., 2018; Talmor and Berant, 2018; Yangetal.,2018),tables(Pasupat and Liang, 2015), and more. Question answering (QA) is commonly used to test the ability to reason, where a complex natural language question is posed, and is to be an- swered given a particular context (text, image, etc.). Although questions often share structure across tasks and modalities, understanding the language of complex questions has thus far been addressed within each task in isolation. Consider the questions in Figure 1, all of which express operations such as fact chaining and counting. Additionally, humans can take a complex question and break it down into a sequence of simpler ques- tions even when they are unaware of what or where the answer is. This ability, to compose and decom- pose questions, lies at the heart of human language (Pelletier, 1994) and allows us to tackle previously unseen problems. Thus, better question under- standing models should improve performance and generalization in tasks that require multi-step rea- soning or that do not have access to substantial amounts of data. In this work we propose question understanding as a standalone language understanding task. We introduce a formalism for representing the mean- ing of questions that relies on question decompo- sition, and is agnostic to the information source. Our formalism, Question Decomposition Meaning Representation (QDMR), is inspired by database query languages (SQL; SPARQL), and by seman- tic parsing (Zelle and Mooney, 1996; Zettlemoyer and Collins, 2005; Clarke et al., 2010), in which questions are given full meaning representations. We express complex questions via simple (‘‘atomic’’) questions that can be executed …

Content truncated to 500 words