A Regularized Competition Model for Question Difficulty Estimation in Community Question Answering Services

Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1115–1126, October 25-29, 2014, Doha, Qatar. c⃝2014 Association for Computational Linguistics A Regularized Competition Model for Question Difficulty Estimation in Community Question Answering Services Quan Wang† Jing Liu‡ Bin Wang† Li Guo† †Institute of Information Engineering, Chinese Academy of Sciences, Beijing, P. R. China {wangquan,wangbin,guoli}@iie.ac.cn ‡Harbin Institute of Technology, Harbin, P. R. China jliu@ir.hit.edu.cn Abstract Estimating questions’ difficulty levels is an important task in community question answering (CQA) services. Previous stud- ies propose to solve this problem based on the question-user comparisons extract- ed from the question answering threads. However, they suffer from data sparseness problem as each question only gets a lim- ited number of comparisons. Moreover, they cannot handle newly posted question- s which get no comparisons. In this pa- per, we propose a novel question difficul- ty estimation approach called Regularized Competition Model (RCM), which natu- rally combines question-user comparisons and questions’ textual descriptions into a unified framework. By incorporating tex- tual information, RCM can effectively deal with data sparseness problem. We further employ a K-Nearest Neighbor approach to estimate difficulty levels of newly post- ed questions, again by leveraging textu- al similarities. Experiments on two pub- licly available data sets show that for both well-resolved and newly-posted question- s, RCM performs the estimation task sig- nificantly better than existing methods, demonstrating the advantage of incorpo- rating textual information. More interest- ingly, we observe that RCM might provide an automatic way to quantitatively mea- sure the knowledge levels of words. 1 Introduction Recent years have seen rapid growth in communi- ty question answering (CQA) services. They have been widely used in various scenarios, including general information seeking on the web1, knowl- 1http://answers.yahoo.com/ edge exchange in professional communities2, and question answering in massive open online cours- es (MOOCs)3, to name a few. An important research problem in CQA is how to automatically estimate the difficulty lev- els of questions, i.e., question difficulty estima- tion (QDE). QDE can benefit many applications. Examples include 1) Question routing. Routing questions to appropriate answerers can help ob- tain quick and high-quality answers (Li and K- ing, 2010; Zhou et al., 2009). Ackerman and McDonald (1996) have demonstrated that rout- ing questions by matching question difficulty lev- el with answerer expertise level will make better use of answerers’ time and expertise. This is even more important for enterprise question answering and MOOCs question answering, where human resources are expensive. 2) Incentive mechanism design. Nam et al. (2009) have found that win- ning point awards offered by reputation system- s is a driving factor for user participation in C- QA services. Assigning higher point awards to more difficult questions will significantly improve user participation and satisfaction. 3) Linguistics analysis. Researchers in computational linguistics are always interested in investigating the correla- tion between language and knowledge, to see how the language reflects one’s knowledge (Church, 2011). As we will show in Section 5.4, QDE pro- vides an automatic way to quantitatively …

Content truncated to 500 words