Cluster-Based Query Expansion for Statistical Question Answering

Cluster-Based Query Expansion for Statistical Question Answering Lucian Vlad Lita ♠ Siemens Medical Solutions lucian.lita@siemens.com Jaime Carbonell Carnegie Mellon University jgc@cs.cmu.edu Abstract Document retrieval is a critical component of question answering (QA), yet little work has been done towards statistical modeling of queries and towards automatic generation of high quality query content for QA. This paper introduces a new, cluster-based query expansion method that learns queries known to be successful when applied to similar questions. We show that cluster-based ex- pansion improves the retrieval performance of a statistical question answering system when used in addition to existing query ex- pansion methods. This paper presents exper- iments with several feature selection meth- ods used individually and in combination. We show that documents retrieved using the cluster-based approach are inherently differ- ent than documents retrieved using existing methods and provide a higher data diversity to answers extractors. 1 Introduction Information retrieval has received sporadic exam- ination in the context of question answering (QA). Over the past several years, research efforts have in- vestigated retrieval quality in very controlled scenar- ios under the question answering task. At a first glance, document and passage retrieval is reason- able when considering the fact that its performance is often above 80% for this stage in the question answering process. However, most often, perfor- mance is measured in terms of the presence of at ♠ work done at Carnegie Mellon least one relevant document in the retrieved docu- ment set, regardless of relevant document density – where a document is relevant if it contains at least one correct answer. More specifically, the retrieval stage is considered successful even if there is a sin- gle document retrieved that mentions a correct an- swer, regardless of context. This performance mea- sure is usually not realistic and revealing in question answering. In typical scenarios, information extraction is not always able to identify correct answers in free text. When successfully found, correct answers are not always assigned sufficiently high confidence scores to ensure their high ranks in the final answer set. As a result, overall question answering scores are still suffering and considerable effort is being di- rected towards improving answer extraction and an- swer merging, yet little attention is being directed towards retrieval. A closer look at retrieval in QA shows that the types of documents retrieved are not always con- ducive to correct answers given existing extraction methods. It is not sufficient to retrieve a relevant document if the answer is difficult to extract from its context. Moreover, the retrieval techniques are often very simple, consisting of extracting keywords from questions, expanding them using conventional meth- ods such as synonym expansion and inflectional ex- pansion, and then running the queries through a re- trieval engine. In order to improve overall question answering performance, additional documents and better doc- uments need to be retrieved. More explicitly, infor- mation retrieval needs to: a) generate query types and query content that is designed to be successful (high precision) for individual questions and …

Content truncated to 500 words