Analyzing the Reading Comprehension Task

Analyzing the Reading Comprehension Task Amit Bagga GE Corporate Research and Development 1 Research Circle Niskayuna, NY 12309 bagga@crd.ge, corn Abstract In this paper we describe a method for analyzing the reading comprehension task. First, we describe a method of classifying facts (information) into cat- egories or levels; where each level signifies a different degree of difficulty of extracting a fact from a piece of text containing it. We then proceed to show how one can use this model the analyze the complexity of the reading comprehension task. Finally, we ana- lyze five different reading comprehension tasks and present results from this analysis. 1 Introduction Recently there has been a spate of activity for build- ing question-answering systems (QA systems) driven largely by the recently organized QA track at the Eighth Text Retrieval Conference (TREC-8) (Har- man, 1999). This increase in research activity has also fueled research in a related area: building Read- ing Comprehension systems (Hirschman and others, 1999). But while a number of successful systems have been developed for each of these tasks, little, if any, work has been done on analyzing the com- plexities of the tasks themselves. In this paper we describe a method of classifying facts (information) into categories or levels; where each level signifies a different degree of difficulty of extracting a fact from a piece of text containing it. We then proceed to show how one can use this model the analyze the complexity of the reading comprehension task. Fi- nally, we analyze five different reading comprehen- sion tasks and present results from this analysis. 2 The Complexity of Extracting a Fact From Text Any text document is a collection of facts (infor- mation). These facts may be explicitly or implicitly stated in the text. In addition, there are "easy" facts which may be found in a single sentence (example: the name of a city) as well as "difficult" facts which are spread across several sentences (example: the reason for a particular event). For a computer system to be able to process text documents in applications like information extrac- 35 tion (IE), question answering, and reading compre- hension, it has to have the ability to extract facts from text. Obviously, the performance of the system will depend upon the type of fact it has to extract: explicit or implicit, easy or difficult, etc. (by no means is this list complete). In addition, the perfor- mance of such systems varies greatly depending on various additional factors including known vocabu- lary, sentence length, the amount of training, quality of parsing, etc. Despite the great variations in the performances of such systems, it has been hypothe- sized that there are facts that are simply harder to extract than others (Hirschman, 1992). In this section we describe a method for estimat- ing the complexity of extracting a fact from text. The proposed model was initially used to analyze the information extraction task (Bagga and Biermann, 1997). In addition to verifying Hirschman's hypoth- esis, the model also provided …

Content truncated to 500 words