Pseudo Relevance Feedback Using Named Entities for Question Answering

Proceedings of the 2006 Australasian Language Technology Workshop (ALTW2006), pages 83–90. Pseudo Relevance Feedback Using Named Entities for Question Answering Luiz Augusto Pizzato and Diego Moll´a CLT - Macquarie University Sydney, Australia {pizzato, diego}@ics.mq.edu.au C´ecile Paris ICT - CSIRO Sydney, Australia Cecile.Paris@csiro.au Abstract Relevance feedback has already proven its usefulness in probabilistic information re- trieval (IR). In this research we explore whether a pseudo relevance feedback tech- nique on IR can improve the Question An- swering task (QA). The basis of our explo- ration is the use of relevant named entities from the top retrieved documents as clues of relevance. We discuss two interesting findings from these experiments: the rea- sons the results were not improved, and the fact that today’s metrics of IR evalu- ation on QA do not reflect the results ob- tained by a QA system. 1 Introduction Probabilistic Information Retrieval estimates the documents’ probability of relevance using a small set of keywords provided by a user. The esti- mation of these probabilities is often assisted by the information contained in documents that are known to be relevant for every specific query. The technique of informing the IR system which doc- uments or information are relevant to a specific query is known as relevance feedback. As reported by Ruthven and Lalmas (2003), relevance feed- back techniques have been used for many years, and they have been shown to improve most proba- bilistic models of Information Retrieval (IR). Relevance feedback is considered as pseudo (or blind) relevance feedback when there is an as- sumption that the top documents retrieved have a higher precision and that their terms represent the subject expected to be retrieved. In other words, it is assumed that the documents on the top of the retrieval list are relevant to the query, and informa- tion from these documents is extracted to generate a new retrieval set. In this paper we explore the use of a pseudo rel- evance feedback technique for the IR stage of a Question Answering (QA) system. It is our under- standing that most questions can be answered us- ing an arbitrary number of documents when query- ing an IR system using the words from the topic and the question. Because QA is normally a com- putationally demanding task, mostly due to the on- line natural language processing tools used, we be- lieve that IR can help QA by providing a small list of high-quality documents, i.e. documents from where the QA system would be able to find the an- swer. In this sense, documents containing answers for a question in a sentence structure that can be easily processed by a QA system will be highly relevant. In this work, we describe an experiment using a pseudo relevance feedback technique applied over a probabilistic IR system to try to improve the per- formance of QA systems. Since most types of factoid questions are answered by named entities, we assume that documents addressing the correct topic but not containing any named entity of …

Content truncated to 500 words