A Probabilistic Framework for Answer Selection in Question Answering

Proceedings of NAACL HLT 2007, pages 524–531, Rochester, NY, April 2007. c⃝2007 Association for Computational Linguistics A Probabilistic Framework for Answer Selection in Question Answering Jeongwoo Ko1, Luo Si2, Eric Nyberg1 1Language Technologies Institute, Carnegie Mellon, Pittsburgh, PA 15213 2Department of Computer Science, Purdue University, West Lafayette, IN 47907 jko@cs.cmu.edu, lsi@cs.purdue.edu, ehn@cs.cmu.edu Abstract This paper describes a probabilistic an- swer selection framework for question an- swering. In contrast with previous work using individual resources such as ontolo- gies and the Web to validate answer can- didates, our work focuses on developing a unified framework that not only uses multiple resources for validating answer candidates, but also considers evidence of similarity among answer candidates in or- der to boost the ranking of the correct an- swer. This framework has been used to se- lect answers from candidates generated by four different answer extraction methods. An extensive set of empirical results based on TREC factoid questions demonstrates the effectiveness of the unified framework. 1 Introduction Question answering aims at finding exact answers to a user’s natural language question from a large collection of documents. Most QA systems com- bine information retrieval with extraction techniques to identify a set of likely candidates and then uti- lize some selection strategy to generate the final answers (Prager et al., 2000; Clarke et al., 2001; Harabagiu et al., 2001). Since answer extractors may be based on imprecise empirical methods, the selection process can be very challenging, as it often entails identifying correct answer(s) amongst many incorrect ones. Question Question Analysis Query Document Retrieval Corpus Docs Answer Extraction Answer candidates Answer Selection Answer Shanghai FT942-2016 0.5 Taiwan FBIS3-45320 0.4 Shanghai FBIS3-58 0.64 Shanghai WSJ920110-0013 0.65 Hong Kong AP880603-0268 0.7 Beijing Document extracted Score Answer candidates Which city in China has the largest number of foreign financial companies? Figure 1: A traditional QA pipeline architecture Figure 1 shows a traditional QA architecture with an example question. Given the question “Which city in China has the largest number of foreign fi- nancial companies?”, the answer extraction com- ponent produces a ranked list of five answer can- didates. Due to imprecision in answer extraction, an incorrect answer (“Beijing”) was ranked at the top position. The correct answer (“Shanghai”) was extracted from two documents with different confi- dence scores and ranked at the third and the fifth po- sitions. In order to select “Shanghai” as the final answer, we need to address two issues: • Answer Validation. How do we identify correct answer(s) amongst incorrect ones? Validating an answer may involve searching for facts in a knowledge base, e.g. IS-A(Shanghai, city), IS-IN(Shanghai, China). • Answer Similarity. How do we exploit evi- dence of similarity among answer candidates? 524 For example, when there are redundant an- swers (“Shanghai”, as above) or several an- swers which represent a single instance (e.g. “Clinton, Bill” and “William Jefferson Clin- ton”) in the candidate list, how much should we boost the answer candidate scores? To address the first issue, several answer selec- tion approaches have used …

Content truncated to 500 words