Question Answering Using Constraint Satisfaction: QA-By-Dossier-With-Contraints

Question Answering using Constraint Satisfaction: QA-by-Dossier-with-Constraints John Prager T.J. Watson Research Ctr. Yorktown Heights N.Y. 10598 jprager@us.ibm.com Jennifer Chu-Carroll T.J. Watson Research Ctr. Yorktown Heights N.Y. 10598 jencc@us.ibm.com Krzysztof Czuba T.J. Watson Research Ctr. Yorktown Heights N.Y. 10598 kczuba@us.ibm.com Abstract QA-by-Dossier-with-Constraints is a new ap- proach to Question Answering whereby candi- date answers’ confidences are adjusted by asking auxiliary questions whose answers con- strain the original answers. These constraints emerge naturally from the domain of interest, and enable application of real-world knowledge to QA. We show that our approach signifi- cantly improves system performance (75% rela- tive improvement in F-measure on select question types) and can create a “dossier” of in- formation about the subject matter in the origi- nal question. 1 Introduction Traditionally, Question Answering (QA) has drawn on the fields of Information Retrieval, Natural Language Processing (NLP), Ontologies, Data Bases and Logical Inference, although it is at heart a prob- lem of NLP. These fields have been used to supply the technology with which QA components have been built. We present here a new methodology which attempts to use QA holistically, along with constraint satisfaction, to better answer questions, without requiring any advances in the underlying fields. Because NLP is still very much an error-prone process, QA systems make many mistakes; accord- ingly, a variety of methods have been developed to boost the accuracy of their answers. Such methods include redundancy (getting the same answer from multiple documents, sources, or algorithms), deep parsing of questions and texts (hence improving the accuracy of confidence measures), inferencing (proving the answer from information in texts plus background knowledge) and sanity-checking (veri- fying that answers are consistent with known facts). To our knowledge, however, no QA system deliber- ately asks additional questions in order to derive constraints on the answers to the original questions. We have found empirically that when our own QA system’s (Prager et al., 2000; Chu-Carroll et al., 2003) top answer is wrong, the correct answer is often present later in the ranked answer list. In other words, the correct answer is in the passages re- trieved by the search engine, but the system was un- able to sufficiently promote the correct answer and/or deprecate the incorrect ones. Our new ap- proach of QA-by-Dossier-with-Constraints (QDC) uses the answers to additional questions to provide more information that can be used in ranking candi- date answers to the original question. These auxil- iary questions are selected such that natural constraints exist among the set of correct answers. After issuing both the original question and auxiliary questions, the system evaluates all possible combi- nations of the candidate answers and scores them by a simple function of both the answers’ intrinsic con- fidences, and how well the combination satisfies the aforementioned constraints. Thus we hope to im- prove the accuracy of an essentially NLP task by making an end-run around some of the more diffi- cult problems in the field. We describe QDC and experiments to evaluate its effectiveness. Our results show that …

Content truncated to 500 words