SemEval-2015 Task 3: Answer Selection in Community Question Answering

Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 269–281, Denver, Colorado, June 4-5, 2015. c⃝2015 Association for Computational Linguistics SemEval-2015 Task 3: Answer Selection in Community Question Answering Preslav Nakov Llu´ıs M`arquez Walid Magdy Alessandro Moschitti ALT Research Group, Qatar Computing Research Institute James Glass MIT Computer Science and Artificial Intelligence Laboratory Bilal Randeree Qatar Living Abstract Community Question Answering (cQA) pro- vides new interesting research directions to the traditional Question Answering (QA) field, e.g., the exploitation of the interaction between users and the structure of related posts. In this context, we organized SemEval- 2015 Task 3 on Answer Selection in cQA, which included two subtasks: (a) classifying answers as good, bad, or potentially relevant with respect to the question, and (b) answering a YES/NO question with yes, no, or unsure, based on the list of all answers. We set subtask A for Arabic and English on two relatively different cQA domains, i.e., the Qatar Liv- ing website for English, and a Quran-related website for Arabic. We used crowdsourcing on Amazon Mechanical Turk to label a large English training dataset, which we released to the research community. Thirteen teams par- ticipated in the challenge with a total of 61 submissions: 24 primary and 37 contrastive. The best systems achieved an official score (macro-averaged F1) of 57.19 and 63.7 for the English subtasks A and B, and 78.55 for the Arabic subtask A. 1 Introduction Many social activities on the Web, e.g., in forums and social networks, are accomplished by means of the community Question Answering (cQA) paradigm. User interaction in this context is seldom moderated, is rather open, and thus has little restric- tions, if any, on who can post and who can answer a question. On the positive side, this means that one can freely ask a question and expect some good, hon- est answers. On the negative side, it takes efforts to go through all possible answers and to make sense of them. It is often the case that many answers are only loosely related to the actual question, and some even change the topic. It is also not unusual for a question to have hundreds of answers, the vast ma- jority of which would not satisfy a user’s informa- tion needs; thus, finding the desired information in a long list of answers might be very time-consuming. In our SemEval-2015 Task 3, we proposed two subtasks. First, subtask A asks for identifying the posts in the answer thread that answer the question well vs. those that can be potentially useful to the user (e.g., because they can help educate him/her on the subject) vs. those that are just bad or useless. This subtask goes in the direction of automating the answer search problem that we discussed above, and we offered it in two languages: English and Ara- bic. Second, for the special case of YES/NO ques- tions, we propose an extreme summarization exer- cise (subtask B), which aims to produce a simple YES/NO overall …

Content truncated to 500 words