An Analysis of the AskMSR Question-Answering System

An Analysis of the AskMSR Question-Answering System Eric Brill, Susan Dumais and Michele Banko Microsoft Research One Microsoft Way Redmond, Wa. 98052 {brill,sdumais,mbanko}@microsoft.com Abstract We describe the architecture of the AskMSR question answering system and systematically evaluate contributions of different system components to accuracy. The system differs from most question answering systems in its dependency on data redundancy rather than sophisticated linguistic analyses of either questions or candidate answers. Because a wrong an- swer is often worse than no answer, we also explore strategies for predicting when the question answering system is likely to give an incorrect answer. 1 Introduction Question answering has recently received attention from the information retrieval, information extrac- tion, machine learning, and natural language proc- essing communities (AAAI, 2002; ACL-ECL, 2002; Voorhees and Harman, 2000, 2001). The goal of a question answering system is to retrieve answers to questions rather than full documents or best-matching passages, as most information re- trieval systems currently do. The TREC Question Answering Track, which has motivated much of the recent work in the field, focuses on fact-based, short-answer questions such as “Who killed Abra- ham Lincoln?” or “How tall is Mount Everest?” In this paper we describe our approach to short answer tasks like these, although the techniques we propose are more broadly applicable. Most question answering systems use a va- riety of linguistic resources to help in understand- ing the user’s query and matching sections in documents. The most common linguistic resources include: part-of-speech tagging, parsing, named entity extraction, semantic relations, dictionaries, WordNet, etc. (e.g., Abney et al., 2000; Chen et al. 2000; Harabagiu et al., 2000; Hovy et al., 2000; Pasca et al., 2001; Prager et al., 2000). We chose instead to focus on the Web as a gigantic data re- pository with tremendous redundancy that can be exploited for question answering. We view our approach as complimentary to more linguistic ap- proaches, but have chosen to see how far we can get initially by focusing on data per se as a key resource available to drive our system design. Re- cently, other researchers have also looked to the web as a resource for question answering (Buch- holtz, 2001; Clarke et al., 2001; Kwok et al., 2001). These systems typically perform complex parsing and entity extraction for both queries and best matching Web pages, and maintain local caches of pages or term weights. Our approach is distinguished from these in its simplicity and effi- ciency in the use of the Web as a large data re- source. Automatic QA from a single, small infor- mation source is extremely challenging, since there is likely to be only one answer in the source to any user’s question. Given a source, such as the TREC corpus, that contains only a relatively small number of formulations of answers to a query, we may be faced with the difficult task of mapping questions to answers by way of uncovering com- plex lexical, syntactic, or semantic relationships between question string and answer string. The …

Content truncated to 500 words