SLS at SemEval-2016 Task 3: Neural-based Approaches for Ranking in Community Question Answering

Proceedings of SemEval-2016, pages 828–835, San Diego, California, June 16-17, 2016. c⃝2016 Association for Computational Linguistics SLS at SemEval-2016 Task 3: Neural-based Approaches for Ranking in Community Question Answering Mitra Mohtarami, Yonatan Belinkov, Wei-Ning Hsu, Yu Zhang Tao Lei, Kfir Bar†, Scott Cyphers, James Glass MIT Computer Science and Artificial Intelligence Laboratory †Tel Aviv University {mitra, belinkov, wnhsu, yzhang87, taolei, cyphers, glass}@csail.mit.edu †kfirbar@post.tau.ac.il Abstract Community question answering platforms need to automatically rank answers and ques- tions with respect to a given question. In this paper, we present the approaches for the An- swer Selection and Question Retrieval tasks of SemEval-2016 (task 3). We develop a bag-of-vectors approach with various vector- and text-based features, and different neu- ral network approaches including CNNs and LSTMs to capture the semantic similarity be- tween questions and answers for ranking pur- pose. Our evaluation demonstrates that our approaches significantly outperform the base- lines. 1 Introduction Community Question Answering (cQA) forums are rapidly growing, resulting in an urgent need to automatically search for relevant answers among many responses provided for a given question (An- swer Selection), and search for relevant questions to reuse their existing answers (Question Retrieval). In this paper, we aim to address the SemEval 2016 tasks (Nakov et al., 2016) that are designed for An- swer Selection (AS) and Question Retrieval (QR). These tasks are briefly described as follows: A Question-Comment Similarity: given a question and its first 10 comments in the question thread, rerank these 10 comments according to their rel- evance with respect to the question. B Question-Question Similarity: given a new question (named original question) and the set of the first 10 related questions retrieved by a search engine, rerank the related questions according to their similarity regarding the original question. C Question-External Comment Similarity: given a new question (original question) and the set of the first 10 related questions retrieved by a search engine, each associated with its first 10 comments appearing in its thread, rerank the 100 comments (10 questions x 10 comments) accord- ing to the new question. D Question-External Question-Comment Pair Similarity: Given a new question and a set of 30 related questions retrieved by a search engine, each associated with one correct answer, rerank the 30 question-comment pairs according to their relevance with respect to the original question. Task B is considered as QR and the others as AS problems. The first three tasks are evaluated on an English dataset and the fourth on an Arabic dataset. Several factors make all these tasks more challeng- ing. First, cQA forums contain open-domain and non-factoid questions and answers, resulting in high variance Q&A quality (M`arquez et al., 2015). A second factor is that the Q&A are long and their length may vary from several words to several hun- dred words. The third factor concerns the relatively close relation between some annotation labels; the comments in the tasks A and C are labeled as Rele- vant, Potential and Irrelevant, and the Relevant com- ments need to be ranked …

Content truncated to 500 words