Flexible Answer Typing with Discriminative Preference Ranking

Proceedings of the 12th Conference of the European Chapter of the ACL, pages 666–674, Athens, Greece, 30 March – 3 April 2009. c⃝2009 Association for Computational Linguistics Flexible Answer Typing with Discriminative Preference Ranking Christopher Pinchak† Dekang Lin‡ Davood Rafiei† †Department of Computing Science ‡Google Inc. University of Alberta 1600 Amphitheatre Parkway Edmonton, Alberta, Canada Mountain View, CA, USA {pinchak,drafiei}@cs.ualberta.ca lindek@google.com Abstract An important part of question answering is ensuring a candidate answer is plausi- ble as a response. We present a flexible approach based on discriminative prefer- ence ranking to determine which of a set of candidate answers are appropriate. Dis- criminative methods provide superior per- formance while at the same time allow the flexibility of adding new and diverse fea- tures. Experimental results on a set of fo- cused What ...? and Which ...? questions show that our learned preference ranking methods perform better than alternative solutions to the task of answer typing. A gain of almost 0.2 in MRR for both the first appropriate and first correct answers is observed along with an increase in pre- cision over the entire range of recall. 1 Introduction Question answering (QA) systems have received a great deal of attention because they provide both a natural means of querying via questions and be- cause they return short, concise answers. These two advantages simplify the task of finding in- formation relevant to a topic of interest. Ques- tions convey more than simply a natural language query; an implicit expectation of answer type is provided along with the question words. The dis- covery and exploitation of this implicit expected type is called answer typing. We introduce an answer typing method that is sufficiently flexible to use a wide variety of fea- tures while at the same time providing a high level of performance. Our answer typing method avoids the use of pre-determined classes that are often lacking for unanticipated answer types. Because answer typing is only part of the QA task, a flexi- ble answer typing model ensures that answer typ- ing can be easily and usefully incorporated into a complete QA system. A discriminative preference ranking model with a preference for appropriate answers is trained and applied to unseen ques- tions. In terms of Mean Reciprocal Rank (MRR), we observe improvements over existing systems of around 0.2 both in terms of the correct answer and in terms of appropriate responses. This increase in MRR brings the performance of our model to near the level of a full QA system on a subset of questions, despite the fact that we rely on answer typing features alone. The amount of information given about the ex- pected answer can vary by question. If the ques- tion contains a question focus, which we define to be the head noun following the wh-word such as city in “What city hosted the 1988 Winter Olympics?”, some of the typing information is ex- plicitly stated. In this instance, the answer is re- quired to be a city. However, there …

Content truncated to 500 words