In Question Answering, Two Heads Are Better Than One

In Question Answering, Two Heads Are Better Than One Jennifer Chu-Carroll Krzysztof Czuba John Prager Abraham Ittycheriah IBM T.J. Watson Research Center P.O. Box 704 Yorktown Heights, NY 10598, U.S.A. jencc,kczuba,jprager,abei@us.ibm.com Abstract Motivated by the success of ensemble methods in machine learning and other areas of natu- ral language processing, we developed a multi- strategy and multi-source approach to question answering which is based on combining the re- sults from different answering agents searching for answers in multiple corpora. The answer- ing agents adopt fundamentally different strate- gies, one utilizing primarily knowledge-based mechanisms and the other adopting statistical techniques. We present our multi-level answer resolution algorithm that combines results from the answering agents at the question, passage, and/or answer levels. Experiments evaluating the effectiveness of our answer resolution algo- rithm show a 35.0% relative improvement over our baseline system in the number of questions correctly answered, and a 32.8% improvement according to the average precision metric. 1 Introduction Traditional question answering (QA) systems typically employ a pipeline approach, consisting roughly of ques- tion analysis, document/passage retrieval, and answer se- lection (see e.g., (Prager et al., 2000; Moldovan et al., 2000; Hovy et al., 2001; Clarke et al., 2001)). Although a typical QA system classifies questions based on expected answer types, it adopts the same strategy for locating po- tential answers from the same corpus regardless of the question classification. In our own earlier work, we de- veloped a specialized mechanism called Virtual Annota- tion for handling definition questions (e.g., “Who was Galileo?” and “What are antibiotics?”) that consults, in addition to the standard reference corpus, a structured knowledge source (WordNet) for answering such ques- tions (Prager et al., 2001). We have shown that better performance is achieved by applying Virtual Annotation and our general purpose QA strategy in parallel. In this paper, we investigate the impact of adopting such a multi- strategy and multi-source approach to QA in a more gen- eral fashion. Our approach to question answering is additionally motivated by the success of ensemble methods in ma- chine learning, where multiple classifiers are employed and their results are combined to produce the final output of the ensemble (for an overview, see (Dietterich, 1997)). Such ensemble methods have recently been adopted in question answering (Chu-Carroll et al., 2003b; Burger et al., 2003). In our question answering system, PI- QUANT, we utilize in parallel multiple answering agents that adopt different processing strategies and consult dif- ferent knowledge sources in identifying answers to given questions, and we employ resolution mechanisms to com- bine the results produced by the individual answering agents. We call our approach multi-strategy since we com- bine the results from a number of independent agents im- plementing different answer finding strategies. We also call it multi-source since the different agents can search for answers in multiple knowledge sources. In this pa- per, we focus on two answering agents that adopt fun- damentally different strategies: one agent uses predomi- nantly knowledge-based mechanisms, whereas the other agent is based on …

Content truncated to 500 words