Question Difficulty Estimation in Community Question Answering Services

Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 85–90, Seattle, Washington, USA, 18-21 October 2013. c⃝2013 Association for Computational Linguistics Question Difficulty Estimation in Community Question Answering Services∗ Jing Liu† Quan Wang‡ Chin-Yew Lin♯ Hsiao-Wuen Hon♯ †Harbin Institute of Technology, Harbin 150001, P.R.China ‡Peking University, Beijing 100871, P.R.China ♯Microsoft Research Asia, Beijing 100080, P.R.China jliu@ir.hit.edu.cn quanwang1012@gmail.com {cyl,hon}@microsoft.com Abstract In this paper, we address the problem of estimating question difficulty in community question answering services. We propose a competition-based model for estimating ques- tion difficulty by leveraging pairwise compar- isons between questions and users. Our ex- perimental results show that our model sig- nificantly outperforms a PageRank-based ap- proach. Most importantly, our analysis shows that the text of question descriptions reflects the question difficulty. This implies the pos- sibility of predicting question difficulty from the text of question descriptions. 1 Introduction In recent years, community question answering (C- QA) services such as Stackoverflow1 and Yahoo! Answers2 have seen rapid growth. A great deal of research effort has been conducted on CQA, in- cluding: (1) question search (Xue et al., 2008; Du- an et al., 2008; Suryanto et al., 2009; Zhou et al., 2011; Cao et al., 2010; Zhang et al., 2012; Ji et al., 2012); (2) answer quality estimation (Jeon et al., 2006; Agichtein et al., 2008; Bian et al., 2009; Liu et al., 2008); (3) user expertise estimation (Jurczyk and Agichtein, 2007; Zhang et al., 2007; Bouguessa et al., 2008; Pal and Konstan, 2010; Liu et al., 2011); and (4) question routing (Zhou et al., 2009; Li and King, 2010; Li et al., 2011). ∗This work was done when Jing Liu and Quan Wang were visiting students at Microsoft Research Asia. Quan Wang is currently affiliated with Institute of Information Engineering, Chinese Academy of Sciences. 1http://stackoverflow.com 2http://answers.yahoo.com However, less attention has been paid to question difficulty estimation in CQA. Question difficulty es- timation can benefit many applications: (1) Experts are usually under time constraints. We do not want to bore experts by routing every question (including both easy and hard ones) to them. Assigning ques- tions to experts by matching question difficulty with expertise level, not just question topic, will make better use of the experts’ time and expertise (Ack- erman and McDonald, 1996). (2) Nam et al. (2009) found that winning the point awards offered by the reputation system is a driving factor in user partici- pation in CQA. Question difficulty estimation would be helpful in designing a better incentive mechanis- m by assigning higher point awards to more diffi- cult questions. (3) Question difficulty estimation can help analyze user behavior in CQA, since users may make strategic choices when encountering questions of different difficulty levels. To the best of our knowledge, not much research has been conducted on the problem of estimating question difficulty in CQA. The most relevant work is a PageRank-based approach proposed by Yang et al. (2008) to estimate task difficulty in crowdsourc- ing contest services. Their key idea is …

Content truncated to 500 words