Answer Attenuation in Question Answering

Answer Attenutation in Question Answering Katie Bell and James R. Curran School of Information Technology University of Sydney Sydney, Australia {kbel5892, james}@it.usyd.edu.au Abstract Research in Question Answering (QA) has been dominated by the TREC methodology of black-box system evaluation. This makes it difficult to evaluate the effectiveness of in- dividual components and requires human in- volvement. We have collected a set of an- swer locations within the AQUAINT corpus for a sample of TREC questions, in doing so we also analyse the ability of humans to retrieve answers. Our answer corpus allows us to track answer attenuation through a QA system. We use this method to evaluate the Pronto QA sys- tem (Bos et al., 2007). 1 Introduction A Question Answering system, is any system which answers questions posed in natural language. In its earliest forms, QA systems were natural lan- guage front-ends to structured databases of knowl- edge (Androutsopoulos, 1995). Today, there exists a massive quantity of data freely available on the in- ternet in raw textual form, but to find specific infor- mation the user may be required to read many doc- uments returned from a query. The goal of open do- main QA is to enable users to find specific answers to questions from enormous corpora spanning many domains. QA can be seen as a search problem: find- ing answers in a corpus of text. A QA system re- duces the search space in stages, starting with se- lecting documents, then passages, and so on, until a single answer is returned. The current method of evaluating Question An- swering systems stems from the Text REtrieval Con- ference (TREC) Question Answering track. Using a standard document collection and question set, each participating system is graded on the propor- tion of answers which are correct and supported by the source document. These criteria are assessed by the only reliable way to determine the correctness and support of an answer: humans. However, man- ual evaluation is costly and not feasible for constant evaluation of QA while developing new systems or techniques. For this purpose a set of correct answer regular expressions are crafted from the correct an- swers as judged by the TREC QA human assessors. These answer keys are unreliable as they are both not exhaustive and do not take into account the sup- port of the source document (Lin, 2005). For reliable automated evaluation we need a gold standard dataset. This would be an exhaustive list of the exact locations of human verified correct answer instances for a set of questions. Using such a set of answer locations, we can define an extension of current evaluation practices, which we call answer attenuation. Answer attenuation is the proportion of correct answers lost from the search space after each stage of processing. The purpose of this mea- sure is both to provide more information about the effectiveness of each system component and a more reliable overall system evaluation. With group of volunteer annotators, we have col- lected an initial test …

Content truncated to 500 words