SemEval-2016 Task 3: Community Question Answering

Proceedings of SemEval-2016, pages 525–545, San Diego, California, June 16-17, 2016. c⃝2016 Association for Computational Linguistics SemEval-2016 Task 3: Community Question Answering Preslav Nakov Llu´ıs M`arquez Alessandro Moschitti Walid Magdy Hamdy Mubarak Abed Alhakim Freihat ALT Research Group, Qatar Computing Research Institute, HBKU James Glass MIT Computer Science and Artificial Intelligence Laboratory Bilal Randeree Qatar Living Abstract This paper describes the SemEval–2016 Task 3 on Community Question Answer- ing, which we offered in English and Ara- bic. For English, we had three sub- tasks: Question–Comment Similarity (subtask A), Question–Question Similarity (B), and Question–External Comment Similarity (C). For Arabic, we had another subtask: Rerank the correct answers for a new question (D). Eighteen teams participated in the task, sub- mitting a total of 95 runs (38 primary and 57 contrastive) for the four subtasks. A variety of approaches and features were used by the participating systems to address the different subtasks, which are summarized in this paper. The best systems achieved an official score (MAP) of 79.19, 76.70, 55.41, and 45.83 in subtasks A, B, C, and D, respectively. These scores are significantly better than those for the baselines that we provided. For subtask A, the best system improved over the 2015 win- ner by 3 points absolute in terms of Accuracy. 1 Introduction Building on the success of SemEval–2015 Task 3 “Answer Selection in Community Question Answer- ing”1 (Nakov et al., 2015), we run an extension in 2016, which covers a full task on Community Question Answering (CQA) and which is, therefore, closer to the real application needs. All the infor- mation related to the task, data, participants, results and publications can be found on the SemEval–2016 Task 3 website.2 1http://alt.qcri.org/semeval2015/task3 2http://alt.qcri.org/semeval2016/task3 CQA forums such as Stack Overflow3 and Qatar Living4, are gaining popularity online. These fo- rums are seldom moderated, quite open, and thus they typically have little restrictions, if any, on who can post and who can answer a question. On the positive side, this means that one can freely ask any question and can then expect some good, honest an- swers. On the negative side, it takes effort to go through all possible answers and to make sense of them. For example, it is not unusual for a ques- tion to have hundreds of answers, which makes it very time-consuming for the user to inspect and to winnow through them all. The present task could help to automate the process of finding good an- swers to new questions in a community-created dis- cussion forum, e.g., by retrieving similar questions in the forum and by identifying the posts in the com- ment threads of those similar questions that answer the original question well. In essence, the main CQA task can be defined as follows: “given (i) a new question and (ii) a large collection of question-comment threads created by a user community, rank the comments that are most useful for answering the new question”. The test question is new with respect to the col- lection, but it is …

Content truncated to 500 words