Learning to Rank Answers on Large Online QA Collections

Proceedings of ACL-08: HLT, pages 719–727, Columbus, Ohio, USA, June 2008. c⃝2008 Association for Computational Linguistics Learning to Rank Answers on Large Online QA Collections Mihai Surdeanu, Massimiliano Ciaramita, Hugo Zaragoza Barcelona Media Innovation Center, Yahoo! Research Barcelona mihai.surdeanu@barcelonamedia.org, {massi,hugo}@yahoo-inc.com Abstract This work describes an answer ranking engine for non-factoid questions built using a large online community-generated question-answer collection (Yahoo! Answers). We show how such collections may be used to effectively set up large supervised learning experiments. Furthermore we investigate a wide range of feature types, some exploiting NLP proces- sors, and demonstrate that using them in com- bination leads to considerable improvements in accuracy. 1 Introduction The problem of Question Answering (QA) has re- ceived considerable attention in the past few years. Nevertheless, most of the work has focused on the task of factoid QA, where questions match short an- swers, usually in the form of named or numerical en- tities. Thanks to international evaluations organized by conferences such as the Text REtrieval Confer- ence (TREC)1 or the Cross Language Evaluation Fo- rum (CLEF) Workshop2, annotated corpora of ques- tions and answers have become available for several languages, which has facilitated the development of robust machine learning models for the task. The situation is different once one moves beyond the task of factoid QA. Comparatively little research has focused on QA models for non-factoid ques- tions such as causation, manner, or reason questions. Because virtually no training data is available for this problem, most automated systems train either 1http://trec.nist.gov 2http://www.clef-campaign.org Q: How do you quiet a squeaky door? A: Spray WD-40 directly onto the hinges of the door. Open and close the door several times. Remove hinges if the door still squeaks. Remove any rust, dirt or loose paint. Apply WD-40 to High removed hinges. Put the hinges back, Quality open and close door several times again. Q: How to extract html tags from an html Low documents with c++? Quality A: very carefully Table 1: Sample content from Yahoo! Answers. on small hand-annotated corpora built in house (Hi- gashinaka and Isozaki, 2008) or on question-answer pairs harvested from Frequently Asked Questions (FAQ) lists or similar resources (Soricut and Brill, 2006). None of these situations is ideal: the cost of building the training corpus in the former setup is high; in the latter scenario the data tends to be domain-specific, hence unsuitable for the learning of open-domain models. On the other hand, recent years have seen an ex- plosion of user-generated content (or social media). Of particular interest in our context are community- driven question-answering sites, such as Yahoo! An- swers3, where users answer questions posed by other users and best answers are selected manually either by the asker or by all the participants in the thread. The data generated by these sites has significant ad- vantages over other web resources: (a) it has a high growth rate and it is already abundant; (b) it cov- ers a large number of topics, hence it offers a better 3http://answers.yahoo.com 719 …

Content truncated to 500 words