A Utility-Driven Approach to Question Ranking in Social QA

Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), pages 125–133, Beijing, August 2010 A Utility-Driven Approach to Question Ranking in Social QA Razvan Bunescu School of EECS Ohio University bunescu@ohio.edu Yunfeng Huang School of EECS Ohio University yh324906@ohio.edu Abstract We generalize the task of finding question paraphrases in a question repository to a novel formulation in which known ques- tions are ranked based on their utility to a new, reference question. We manually annotate a dataset of 60 groups of ques- tions with a partial order relation reflect- ing the relative utility of questions inside each group, and use it to evaluate mean- ing and structure aware utility functions. Experimental evaluation demonstrates the importance of using structural informa- tion in estimating the relative usefulness of questions, holding the promise of in- creased usability for social QA sites. 1 Introduction Open domain Question Answering (QA) is one of the most complex and challenging tasks in natural language processing. While building on ideas from Information Retrieval (IR), question answering is generally seen as a more difficult task due to constraints on both the input represen- tation (natural language questions vs. keyword- based queries) and the form of the output (fo- cused answers vs. entire documents). Recently, community-driven QA sites such as Yahoo! An- swers and WikiAnswers have established a new approach to question answering in which the bur- den of dealing with the inherent complexity of open domain QA is shifted from the computer system to volunteer contributors. The computer is no longer required to perform a deep linguis- tic analysis of questions and generate correspond- ing answers, and instead acts as a mediator be- tween users submitting questions and volunteers providing the answers. In most implementations of community-driven QA, the mediator system has a well defined strategy for enticing volun- teers to post high quality answers on the website. In general, the overall objective is to minimize the response time and maximize the accuracy of the answers, measures that are highly correlated with user satisfaction. For any submitted ques- tion, one useful strategy is to search the QA repos- itory for similar questions that have already been answered, and provide the corresponding ranked list of answers, if such a question is found. The success of this approach depends on the definition and implementation of the question-to-question similarity function. In the simplest solution, the system searches for previously answered ques- tions based on exact string matching with the reference question. Alternatively, sites such as WikiAnswers allow the users to mark questions they think are rephrasings (“alternate wordings”, or paraphrases) of existing questions. These ques- tion clusters are then taken into account when per- forming exact string matching, therefore increas- ing the likelihood of finding previously answered questions that are semantically equivalent to the reference question. Like the original question an- swering task, the solution to question rephrasing is also based on volunteer contributions. In order to lessen the amount of work required from the con- tributors, an alternative solution …

Content truncated to 500 words