Exploiting Salient Patterns for Question Detection and Question Retrieval in Community-based Question Answering

Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), pages 1155–1163, Beijing, August 2010 Exploiting Salient Patterns for Question Detection and Question Retrieval in Community-based Question Answering Kai Wang Department of Computer Science School of Computing National University of Singapore kwang@comp.nus.edu.sg Tat-Seng Chua Department of Computer Science School of Computing National University of Singapore chuats@comp.nus.edu.sg Abstract Question detection serves great purposes in the cQA question retrieval task. While detecting questions in standard language data corpus is relatively easy, it becomes a great challenge for online content. On- line questions are usually long and infor- mal, and standard features such as ques- tion mark or 5W1H words are likely to be absent. In this paper, we explore ques- tion characteristics in cQA services, and propose an automated approach to detect question sentences based on lexical and syntactic features. Our model is capable of handling informal online languages. The empirical evaluation results further demonstrate that our model significantly outperforms traditional methods in de- tecting online question sentences, and it considerably boosts the question retrieval performance in cQA. 1 Introduction Community-based Question Answering services (cQA) such as Yahoo! Answers have emerged as popular means of information exchange on the web. They not only connect a network of people to freely ask and answer questions, but also allow information seekers to search for relevant histori- cal questions in the cQA archive (Agichtein et al., 2008; Xue et al., 2008; Wang et al., 2009). Many research works have been proposed to find similar questions in cQA. The state-of-the-art retrieval models include the vector space model (Duan et al., 2008), language model (Duan et al., 2008; Jeon et al., 2005), Okapi model (Jeon et al., 2005), translation model (Jeon et al., 2005; Rie- zler et al., 2007; Xue et al., 2008), and syntac- tic tree matching model(Wang et al., 2009). Al- though experimental studies in these works show that the proposed models are capable of improv- ing question retrieval, they did not give clear ex- planation on which portion of the question that the user query is actually matched against. question thread from cQA usually comprises sev-A eral sub-questions conveying different informa- tion needs, and it is highly desirable to identify individual sub-questions and match each of them to the user query. Getting sub-questions clearly identified not only helps the retrieval system to match user query to the most desirable content but also improves the retrieval efficiency. However, the detection of sub-question is non- trivial. Question sentences in cQA are usually mixed with various description sentences, and they usually employ informal languages, where standard features such as question mark or ut- terance are likely to be absent. As such, simple heuristics using question mark or 5W1H words (who, what, where, why, how) may become in- adequate. The demand of special techniques in detecting question sentences online arises due to three particular reasons. First, the question mark could be missing at the end of a question1, or might be used in cases other than questions …

Content truncated to 500 words