SemEval-2017 Task 3: Community Question Answering

Proceedings of the 11th International Workshop on Semantic Evaluations (SemEval-2017), pages 27–48, Vancouver, Canada, August 3 - 4, 2017. c⃝2017 Association for Computational Linguistics SemEval-2017 Task 3: Community Question Answering Preslav Nakov1 Doris Hoogeveen2 Llu´ıs M`arquez1 Alessandro Moschitti1 Hamdy Mubarak1 Timothy Baldwin2 Karin Verspoor2 1ALT Research Group, Qatar Computing Research Institute, HBKU 2The University of Melbourne Abstract We describe SemEval2017 Task 3 on Com- munity Question Answering. This year, we reran the four subtasks from SemEval- 2016: (A) Question–Comment Similarity, (B) Question–Question Similarity, (C) Question– External Comment Similarity, and (D) Rerank the correct answers for a new question in Arabic, providing all the data from 2015 and 2016 for training, and fresh data for testing. Additionally, we added a new subtask E in order to enable ex- perimentation with Multi-domain Question Du- plicate Detection in a larger-scale scenario, using StackExchange subforums. A total of 23 teams participated in the task, and submitted a total of 85 runs (36 primary and 49 contrastive) for sub- tasks A–D. Unfortunately, no teams participated in subtask E. A variety of approaches and fea- tures were used by the participating systems to address the different subtasks. The best systems achieved an official score (MAP) of 88.43, 47.22, 15.46, and 61.16 in subtasks A, B, C, and D, re- spectively. These scores are better than the base- lines, especially for subtasks A–C. 1 Introduction Community Question Answering (CQA) on web forums such as Stack Overflow1 and Qatar Liv- ing,2 is gaining popularity, thanks to the flexibility of forums to provide information to a user (Mos- chitti et al., 2016). Forums are moderated only in- directly via the community, rather open, and sub- ject to few restrictions, if any, on who can post and answer a question, or what questions can be asked. On the positive side, a user can freely ask any question and can expect a variety of answers. On the negative side, it takes efforts to go through the provided answers of varying quality and to make sense of them. It is not unusual for a popular ques- tion to have hundreds of answers, and it is very time-consuming for a user to inspect them all. 1http://stackoverflow.com/ 2http://www.qatarliving.com/forum Hence, users can benefit from automated tools to help them navigate these forums, including sup- port for finding similar existing questions to a new question, and for identifying good answers, e.g., by retrieving similar questions that already provide an answer to the new question. Given the important role that natural language processing (NLP) plays for CQA, we have orga- nized a challenge series to promote related re- search for the past three years. We have provided datasets, annotated data and we have developed robust evaluation procedures in order to establish a common ground for comparing and evaluating different approaches to CQA. In greater detail, in SemEval-2015 Task 3 “An- swer Selection in Community Question Answer- ing” (Nakov et al., 2015),3 we mainly targeted conventional Question Answering (QA) tasks, i.e., answer selection. In contrast, in SemEval- 2016 …

Content truncated to 500 words