Deep Read: A Reading Comprehension System

Deep Read: A Reading Comprehension System Lynette Hirschman • Marc Light • Eric Breck • John D. Burger The MITRE Corporation 202 Burlington Road Bedford, MA USA 01730 { l ynette, light, ebreck, john } @ mitre.org Abstract This paper describes initial work on Deep Read, an automated reading comprehension system that accepts arbitrary text input (a story) and answers questions about it. We have acquired a corpus of 60 development and 60 test stories of 3 rd to 6 th grade material; each story is followed by short-answer questions (an answer key was also provided). We used these to construct and evaluate a baseline system that uses pattern matching (bag-of-words) techniques augmented with additional automated linguistic processing (stemming, name identification, semantic class identification, and pronoun resolution). This simple system retrieves the sentence containing the answer 30-40% of the time. 1 Introduction This paper describes our initial work exploring reading comprehension tests as a research problem and an evaluation method for language understanding systems. Such tests can take the form of standardized multiple-choice diagnostic reading skill tests, as well as fill-in- the-blank and short-answer tests. Typically, such tests ask the student to read a story or article and to demonstrate her/his understanding of that article by answering questions about it. For an example, see Figure 1. Reading comprehension tests are interesting because they constitute "found" test material: these tests are created in order to evaluate children's reading skills, and therefore, test materials, scoring algorithms, and human performance measures already exist. Furthermore, human performance measures provide a more intuitive way of assessing the capabilities of a given system than current measures of precision, recall, F-measure, operating curves, etc. In addition, reading comprehension tests are written to test a range of skill levels. With proper choice of test material, it should be possible to challenge systems to successively higher levels of performance. For these reasons, reading comprehension tests offer an interesting alternative to the kinds of special-purpose, carefully constructed evaluations that have driven much recent research in language understanding. Moreover, the current state-of-the- art in computer-based language understanding makes this project a good choice: it is beyond current systems' capabilities, but tractable. Our Library of Congress Has Books for Everyone (WASHINGTON, D.C., 1964) - It was 150 years ago this year that our nation's biggest library burned to the ground. Copies of all the wriuen books of the time were kept in the Library of Congress. But they were destroyed by fire in 1814 during a war with the British. That fire didn't stop book lovers. The next year, they began to rebuild the library. By giving it 6,457 of his books, Thomas Jefferson helped get it started. The first libraries in the United States could be used by members only. But the Library of Congress was built for all the people. From the start, it was our national library. Today, the Library of Congress is one of the largest libraries in the world. People can find a copy of just about …

Content truncated to 500 words