An Evaluation on Query-biased Summarisation for the Question Answering Task

Evaluation of a Query-biased Document Summarisation Approach for the Question Ansnwering Task Mingfang Wu1 Ross Wilkinson1 Cécile Paris2 CSIRO ICT Center, 1Gate 7, 71 Normanby Road, Private Bag 10, South Clayton 3169 2Locked Bag 17, North Ryde, NSW 1670, Australia {Mingfang.Wu, Ross.Wilkinson, Cecile.Paris}@csiro.au Abstract This paper presents an approach on a query- biased document summarisation and an evaluation of the use of such an approach for question answering tasks. We observed a significant difference in the user’s performance by presenting a list of documents customized to the task type, compared with a generic document summarization approach. This indicates that paying attention to the task and the searcher interaction may provide substantial improvement in task performance. 1 Introduction People are searching information to meet their information needs from their tasks at hand. However, most search engines interact with user in a “one size fits all” fashion and ignore the user’s preferences, search context or the task context. The burden is then placed on the user to scan, navigate, and read the retrieved documents to identify what s/he wants. We believe that paying attention to the nature of the information task and the needs of the searcher may provide benefits beyond those available through more accurate matching. As Saracevic [9] pointed out, the key to the future of information systems and searching processes lies not only in increased sophistication of technology, but also in increased understanding of human involvement with information. In the study presented in this paper, we examine searchers’ ability to carry out a question answering task [13]. Unlike the task of the non-interactive TREC question answer track [10], where the question answering focuses on fact-based, short answer questions such as “Who is the first prime minister of Australia”. We looked at the type of question answering task that is more complex than the task of finding a single fact. The answer to this type of questions would not generally be available in a single document, but would require facts to be extracted from several documents. For example, an Australian cattle farmer would like an information access system that could tell s/he “which countries are the top ten importers of Australian beef?”. An ideal answer should consist of a list of country names together with corresponding beef import data. This answer could be synthesized from scattered information collected from various sources, such as a news article about Japanese meat imports and an analysis report on Australian beef in the European market. The successful completion of such a task requires an answer to be obtained, citing the relevant source documents. If we assume that we do not have an advanced language engine that can understand such questions and then synthesize answers to them, a searcher will be involved in the process, beyond simply initiating a query and reading a list of answers. Some of the elements that might lead to successful answering might • include: support for query formulation (and re- formulation) • effective matching and ordering of candidate documents • …

Content truncated to 500 words