Examining the Role of Statistical and Linguistic Knowledge Sources in a General-Knowledge Question-Answering System

Examining the Role of Statistical and Linguistic Knowledge Sources in a General-Knowledge Question-Answering System Claire Cardie 1 and Vincent Ng 1 and David Pierce 1 and Chris Buckley 2 Department of Computer Science, Cornell University, Ithaca, NY 148531 SaBIR Research 2 E-mail: cardie,yung,pierce@cs.cornell.edu, chrisb@sabir.com Abstract We describe and evaluate an implemented system for general-knowledge question answering. The sys- tem combines techniques for standard ad-hoc infor- mation retrieval (IR), query-dependent text summa- rization, and shallow syntactic and semantic sen- tence analysis. In a series of experiments we examine the role of each statistical and linguistic knowledge source in the question-answering system. In con- trast to previous results, we find first that statisti- cal knowledge of word co-occurrences as computed by IR vector space methods can be used to quickly and accurately locate the relevant documents for each question. The use of query-dependent text summarization techniques, however, provides only small increases in performance and severely limits recall levels when inaccurate. Nevertheless, it is the text summarization component that allows subse- quent linguistic filters to focus on relevant passages. We find that even very weak linguistic knowledge can offer substantial improvements over purely IR- based techniques for question answering, especially when smoothly integrated with statistical prefer- ences computed by the IR subsystems. 1 Introduction In this paper, we describe and evaluate an imple- mented system for general-knowledge question an- swering. Open-ended question-answering systems that allow users to pose a question of any type, in any language, without domain restrictions, remain beyond the scope of today's text-processing systems. We investigate instead a restricted, but nevertheless useful variation of the problem (TREC-8, 2000): Given a large text collection and a set of questions specified in English, find answers to the questions in the collection. In addition, the restricted task guarantees that: • the answer exists in the collection, • all supporting information for the answer lies in a single document, and • the answer is short m less than 50 bytes in length. Consider, for example, the question Which country has the largest part of the Amazon rain forest?, taken from the TREC8 Question Answering development corpus. The answer (in document LA032590-0089) is Brazil Previous research has addressed similar question- answering (QA) scenarios using a variety of natu- ral language processing (NLP) and information re- trieval (IR) techniques. Lehnert (1978) tackles the difficult task of answering questions in the context of story understanding. Unlike our restricted QA task, questions to Lehnert's system often require answers that are not explicitly mentioned in the story. Her goal then is to answer questions by making infer- ences about actions and actors in the story using world knowledge in the form of scripts, plans, and goals (Schank and Abelson, 1977). More recently, Burke et al. (1995; 1997) describe a system that an- swers natural language questions using a database of question-answer pairs built from existing frequently- asked question (FAQ) files. Their FAQFinder sys- tem uses IR techniques to match the given question to questions in the database. It then …

Content truncated to 500 words