SQuAD: 100,000+ Questions for Machine Comprehension of Text

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics SQuAD: 100,000+ Questions for Machine Comprehension of Text Pranav Rajpurkar and Jian Zhang and Konstantin Lopyrev and Percy Liang {pranavsr,zjian,klopyrev,pliang}@cs.stanford.edu Computer Science Department Stanford University Abstract We present the Stanford Question Answer- ing Dataset (SQuAD), a new reading compre- hension dataset consisting of 100,000+ ques- tions posed by crowdworkers on a set of Wikipedia articles, where the answer to each question is a segment of text from the cor- responding reading passage. We analyze the dataset to understand the types of reason- ing required to answer the questions, lean- ing heavily on dependency and constituency trees. We build a strong logistic regression model, which achieves an F1 score of 51.0%, a significant improvement over a simple base- line (20%). However, human performance (86.8%) is much higher, indicating that the dataset presents a good challenge problem for future research. The dataset is freely available at https://stanford-qa.com. 1 Introduction Reading Comprehension (RC), or the ability to read text and then answer questions about it, is a chal- lenging task for machines, requiring both under- standing of natural language and knowledge about the world. Consider the question “what causes pre- cipitation to fall?” posed on the passage in Figure 1. In order to answer the question, one might first lo- cate the relevant part of the passage “precipitation ... falls under gravity”, then reason that “under” refers to a cause (not location), and thus determine the cor- rect answer: “gravity”. How can we get a machine to make progress on the challenging task of reading comprehension? Historically, large, realistic datasets have played In meteorology, precipitation is any product of the condensation of atmospheric water vapor that falls under gravity. The main forms of pre- cipitation include drizzle, rain, sleet, snow, grau- pel and hail... Precipitation forms as smaller droplets coalesce via collision with other rain drops or ice crystals within a cloud. Short, in- tense periods of rain in scattered locations are called “showers”. What causes precipitation to fall? gravity What is another main form of precipitation be- sides drizzle, rain, snow, sleet and hail? graupel Where do water droplets collide with ice crystals to form precipitation? within a cloud Figure 1: Question-answer pairs for a sample passage in the SQuAD dataset. Each of the answers is a segment of text from the passage. a critical role for driving fields forward—famous examples include ImageNet for object recognition (Deng et al., 2009) and the Penn Treebank for syntactic parsing (Marcus et al., 1993). Existing datasets for RC have one of two shortcomings: (i) those that are high in quality (Richardson et al., 2013; Berant et al., 2014) are too small for training modern data-intensive models, while (ii) those that are large (Hermann et al., 2015; Hill et al., 2015) are semi-synthetic and do not share the same character- istics as explicit reading comprehension questions. To address the need for a …

Content truncated to 500 words