DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs

Proceedings of NAACL-HLT 2019, pages 2368–2378 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 2368 DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs Dheeru Dua♣, Yizhong Wang♦∗, Pradeep Dasigi♥, Gabriel Stanovsky♥+, Sameer Singh♣, and Matt Gardner♠ ♣University of California, Irvine, USA ♦Peking University, Beijing, China ♥Allen Institute for Artificial Intelligence, Seattle, Washington, USA ♠Allen Institute for Artificial Intelligence, Irvine, California, USA +University of Washington, Seattle, Washington, USA ddua@uci.edu Abstract Reading comprehension has recently seen rapid progress, with systems matching humans on the most popular datasets for the task. How- ever, a large body of work has highlighted the brittleness of these systems, showing that there is much work left to be done. We in- troduce a new English reading comprehension benchmark, DROP, which requires Discrete Reasoning Over the content of Paragraphs. In this crowdsourced, adversarially-created, 96k- question benchmark, a system must resolve references in a question, perhaps to multiple in- put positions, and perform discrete operations over them (such as addition, counting, or sort- ing). These operations require a much more comprehensive understanding of the content of paragraphs than what was necessary for prior datasets. We apply state-of-the-art methods from both the reading comprehension and se- mantic parsing literatures on this dataset and show that the best systems only achieve 32.7% F1 on our generalized accuracy metric, while expert human performance is 96.4%. We ad- ditionally present a new model that combines reading comprehension methods with simple numerical reasoning to achieve 47.0% F1. 1 Introduction The task of reading comprehension, where sys- tems must understand a single passage of text well enough to answer arbitrary questions about it, has seen significant progress in the last few years, so much that the most popular datasets available for this task have been solved (Chen et al., 2016; De- vlin et al., 2019). We introduce a substantially more challenging English reading comprehension dataset aimed at pushing the field towards more comprehensive analysis of paragraphs of text. In ∗Work done as an intern at the Allen Institute for Artificial Intelligence in Irvine, California. this new benchmark, which we call DROP, a sys- tem is given a paragraph and a question and must perform some kind of Discrete Reasoning Over the text in the Paragraph to obtain the correct answer. These questions that require discrete reasoning (such as addition, sorting, or counting; see Table 1) are inspired by the complex, compositional ques- tions commonly found in the semantic parsing lit- erature. We focus on this type of questions because they force a structured analysis of the content of the paragraph that is detailed enough to permit reason- ing. Our goal is to further paragraph understand- ing; complex questions allow us to test a system’s understanding of the paragraph’s semantics. DROP is also designed to further research on methods that combine distributed representations with symbolic, discrete reasoning. In order to do well on this dataset, a system must be able to find multiple occurrences of an event described in …

Content truncated to 500 words