Improving QA Accuracy by Question Inversion

Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the ACL, pages 1073–1080, Sydney, July 2006. c⃝2006 Association for Computational Linguistics Improving QA Accuracy by Question Inversion John Prager IBM T.J. Watson Res. Ctr. Yorktown Heights N.Y. 10598 jprager@us.ibm.com Pablo Duboue IBM T.J. Watson Res. Ctr. Yorktown Heights N.Y. 10598 duboue@us.ibm.com Jennifer Chu-Carroll IBM T.J. Watson Res. Ctr. Yorktown Heights N.Y. 10598 jencc@us.ibm.com Abstract This paper demonstrates a conceptually simple but effective method of increasing the accuracy of QA systems on factoid-style questions. We define the notion of an inverted question, and show that by requiring that the answers to the original and inverted questions be mutually con- sistent, incorrect answers get demoted in confi- dence and correct ones promoted. Additionally, we show that lack of validation can be used to assert no-answer (nil) conditions. We demon- strate increases of performance on TREC and other question-sets, and discuss the kinds of fu- ture activities that can be particularly beneficial to approaches such as ours. 1 Introduction Most QA systems nowadays consist of the following standard modules: QUESTION PROCESSING, to de- termine the bag of words for a query and the desired answer type (the type of the entity that will be of- fered as a candidate answer); SEARCH, which will use the query to extract a set of documents or pas- sages from a corpus; and ANSWER SELECTION, which will analyze the returned documents or pas- sages for instances of the answer type in the most favorable contexts. Each of these components im- plements a set of heuristics or hypotheses, as de- vised by their authors (cf. Clarke et al. 2001, Chu- Carroll et al. 2003). When we perform failure analysis on questions in- correctly answered by our system, we find that there are broadly speaking two kinds of failure. There are errors (we might call them bugs) on the implementa- tion of the said heuristics: errors in tagging, parsing, named-entity recognition; omissions in synonym lists; missing patterns, and just plain programming errors. This class can be characterized by being fix- able by identifying incorrect code and fixing it, or adding more items, either explicitly or through train- ing. The other class of errors (what we might call unlucky) are at the boundaries of the heuristics; situations were the system did not do anything “wrong,” in the sense of bug, but circumstances con- spired against finding the correct answer. Usually when unlucky errors occur, the system gen- erates a reasonable query and an appropriate answer type, and at least one passage containing the right answer is returned. However, there may be returned passages that have a larger number of query terms and an incorrect answer of the right type, or the query terms might just be physically closer to the incorrect answer than to the correct one. ANSWER SELECTION modules typically work either by trying to prove the answer is correct (Moldovan & Rus, 2001) or by giving them a weight produced by summing a …

Content truncated to 500 words