Cross-Domain Answer Ranking using Importance Sampling

International Joint Conference on Natural Language Processing, pages 987–991, Nagoya, Japan, 14-18 October 2013. Cross-domain answer ranking using importance sampling Anders Johannsen Anders Søgaard University of Copenhagen {ajohannsen,soegaard}@hum.ku.dk Abstract We consider the problem of learning how to rank answers across domains in com- munity question answering using stylistic features. Our main contribution is an im- portance sampling technique for selecting training data per answer thread. Our ap- proach is evaluated across 30 community sites and shown to be significantly better than random sampling. We show that the most useful features in our model relate to answer length and overlap with question. 1 Introduction Community Q&A (cQA) sites are rich sources of knowledge, offering information often not avail- able elsewhere. While questions often attract the attention of experts, anyone can chip in, and as a result answer quality varies a lot (Fichman, 2011). cQA sites deal with this problem by engaging the users. If people like an answer or find it useful, they vote it up, and if it is wrong, unhelpful or spammy, it gets a down vote and is sometimes re- moved altogether. To a large degree the success of cQA can be attributed to this powerful content filtering mechanism. The voting induces a ranking of the answers, and that is the ranking we wish to reproduce in this paper. We are interested in learning a ranking model based on textual or stylistic features only, ex- tracted from the question and the answer candi- date, because willfully ignoring information about user behavior and other social knowledge avail- able in cQA sites makes our model applicable in a wider range of circumstances. Outside the world of cQA, automatic answer ranking might, for in- stance, be used to prioritize lists of answers found in FAQs or embedded in running text. In other words, we are interested in learning a reranking model that is generally applicable to question an- swering systems. Part of what makes one answer preferable to another is how effective it is in communicating its advice. There may be plenty of answers that in some technical sense are correct and yet are not especially helpful. For instance, if the kind of advice we are looking for involves a proce- dure, an answer structured as “First ... Then ... Finally” would probably be of greater use to us than an answer with no discernible temporal struc- ture. Our features capture aspects of the discourse surface structure of the answer. If the model is supposed to be generally applicable to question answering it also needs to exhibit robust perfor- mance across domains. Learning that mentions of specific Python modules correlate with answer quality in Stack Overflow does not help us an- swer questions in the cooking domain. We need to limit ourselves to features that transfer across domains. We further hypothesize a link between question type and answer structure (e.g. good answers to how-to questions look different from good answers to questions that ask for definitions), and test this experimentally by …

Content truncated to 500 words