KeLP at SemEval-2016 Task 3: Learning Semantic Relations between Questions and Answers

Proceedings of SemEval-2016, pages 1116–1123, San Diego, California, June 16-17, 2016. c⃝2016 Association for Computational Linguistics KeLP at SemEval-2016 Task 3: Learning Semantic Relations between Questions and Answers Simone Filice1, Danilo Croce2, Alessandro Moschitti3 and Roberto Basili2 1 DICII, University of Roma, Tor Vergata 2 DII, University of Roma, Tor Vergata 3 ALT, Qatar Computing Research Institute, HBKU {filice,croce,basili}@info.uniroma2.it amoschitti@qf.org.qa Abstract This paper describes the KeLP system par- ticipating in the SemEval-2016 Community Question Answering (cQA) task. The chal- lenge tasks are modeled as binary classifi- cation problems: kernel-based classifiers are trained on the SemEval datasets and their scores are used to sort the instances and pro- duce the final ranking. All classifiers and kernels have been implemented within the Kernel-based Learning Platform called KeLP. Our primary submission ranked first in Sub- task A, third in Subtask B and second in Sub- task C. These ranks are based on MAP, which is the referring challenge system score. Our approach outperforms all the other systems with respect to all the other challenge metrics. 1 Introduction This paper describes the KeLP system participating in the SemEval-2016 cQA challenge. In this task, participants are asked to automatically provide good answers in a cQA setting (Nakov et al., 2016). In particular, the main task is: given a new question and a large collection of question-comment threads created by a user community, rank the most useful comments on the top. We participated in all English subtasks: the datasets were extracted from Qatar Living1, a web forum where people pose questions about multiple aspects of their daily life in Qatar. Three subtasks are associated with the English challenge: Subtask A: Given a question q and its first 10 comments c1, . . . , c10 in its question thread, re-rank 1http://www.qatarliving.com/forum these 10 comments according to their relevance with respect to the question, i.e., the good comments have to be ranked above potential or bad comments. Subtask B: Given a new question o and the set of the first 10 related questions q1, . . . , q10 (retrieved by a search engine), re-rank the related questions accord- ing to their similarity with respect to o, i.e., the per- fect match and relevant questions should be ranked above the irrelevant ones. Subtask C: Given a new question o, and the set of the first 10 related questions, q1, . . . , q10, (retrieved by a search engine), each one associated with its first 10 comments, cq 1, . . . , cq 10, appearing in its thread, re-rank the 100 comments according to their rele- vance with respect to o, i.e., the good comments are to be ranked above potential or bad comments. All the above subtasks have been modeled as bi- nary classification problems: kernel-based classi- fiers are trained and the classification score is used to sort the instances and produce the final ranking. All classifiers and kernels have been implemented within the Kernel-based Learning Platform2 (KeLP) (Filice et al., 2015b), thus determining the …

Content truncated to 500 words