The Use of Metadata, Web-derived Answer Patterns and Passage Context to Improve Reading Comprehension Performance

Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing (HLT/EMNLP), pages 604–611, Vancouver, October 2005. c⃝2005 Association for Computational Linguistics The Use of Metadata, Web-derived Answer Patterns and Passage Context to Improve Reading Comprehension Performance Yongping Du Media Computing and Web Intelligence Laboratory Fudan University Shanghai, China ypdu@fudan.edu.cn Helen Meng Human-Computer Communication Laboratory The Chinese University of Hong Kong HongKong. SAR. China hmmeng@se.cuhk.edu.hk Xuanjing Huang Media Computing and Web Intelligence Laboratory Fudan University Shanghai, China xjhuang@fudan.edu.cn Lide Wu Media Computing and Web Intelligence Laboratory Fudan University Shanghai, China ldwu@fudan.edu.cn Abstract A reading comprehension (RC) system attempts to understand a document and returns an answer sentence when posed with a question. RC resembles the ad hoc question answering (QA) task that aims to extract an answer from a collection of documents when posed with a question. However, since RC focuses only on a single document, the system needs to draw upon external knowledge sources to achieve deep analysis of passage sentences for answer sentence extraction. This paper proposes an approach towards RC that attempts to utilize external knowledge to improve performance beyond the baseline set by the bag-of-words (BOW) approach. Our approach emphasizes matching of metadata (i.e. verbs, named entities and base noun phrases) in passage context utilization and answer sentence extraction. We have also devised an automatic acquisition process for Web-derived answer patterns (AP) which utilizes question-answer pairs from TREC QA, the Google search engine and the Web. This approach gave improved RC performances for both the Remedia and ChungHwa corpora, attaining HumSent accuracies of 42% and 69% respectively. In particular, performance analysis based on Remedia shows that relative performances of 20.7% is due to metadata matching and a further 10.9% is due to the application of Web-derived answer patterns. 1. Introduction A reading comprehension (RC) system attempts to understand a document and returns an answer sentence when posed with a question. The RC task was first proposed by the MITRE Corporation which developed the Deep Read reading comprehension system (Hirschman et al., 1999). Deep Read was evaluated on the Remedia Corpus that contains a set of stories, each with an average of 20 sentences and five questions (of types who, where, when, what and why). The MITRE group also defined the HumSent scoring metric, i.e. the percentage of test questions for which the system has chosen a correct sentence as the answer. HumSent answers were compiled by a human annotator, who examined the stories and chose the sentence(s) that best answered the questions. It was judged that for 11% of the Remedia test questions, there is no single sentence in the story that is judged to be an appropriate answer sentence. Hence the upper bound for RC on Remedia should by 89% HumSent accuracy. (Hirschman et al. 1999) reported a HumSent accuracy of 36.6% on the Remedia test set. Subsequently, (Ng et al., 2000) used a machine learning approach of decision tree and achieved the accuracy of 39.3%. Then (Riloff and Thelen, 2000) …

Content truncated to 500 words