SUper Team at SemEval-2016 Task 3: Building a Feature-Rich System for Community Question Answering

Proceedings of SemEval-2016, pages 836–843, San Diego, California, June 16-17, 2016. c⃝2016 Association for Computational Linguistics SUper Team at SemEval-2016 Task 3: Building a Feature-Rich System for Community Question Answering Tsvetomila Mihaylova, Pepa Gencheva, Martin Boyanov, Ivana Yovcheva, Todor Mihaylov, Momchil Hardalov, Yasen Kiprov, Daniel Balchev, Ivan Koychev FMI, Sofia University “St. Kliment Ohridski”, Sofia, Bulgaria Preslav Nakov ALT Research Group, Qatar Computing Research Institute, HBKU, Doha, Qatar Ivelina Nikolova, Galia Angelova IICT, Bulgarian Academy of Sciences, Sofia, Bulgaria (iva@lml.bas.bg) Abstract We present the system we built for partici- pating in SemEval-2016 Task 3 on Commu- nity Question Answering. We achieved the best results on subtask C, and strong results on subtasks A and B, by combining a rich set of various types of features: semantic, lexical, metadata, and user-related. The most impor- tant group turned out to be the metadata for the question and for the comment, semantic vectors trained on QatarLiving data and simi- larities between the question and the comment for subtasks A and C, and between the original and the related question for Subtask B. 1 Introduction SemEval-2016 Task 3 on Community Question An- swering1 (Nakov et al., 2016) aims to solve a real- life application problem. The main subtask C (Question-External Comment Similarity) asks to find an answer in the forum that will be appropriate as a response to a newly posted question. This is achieved by retrieving similar questions and ranking their answers with respect to the new question. Two additional supporting subtasks are defined: Subtask A (Question-Comment Similarity): Given a question from a question-comment thread, rank the comments within the thread based on their relevance with respect to the question. Subtask B (Question-Question Similarity): Given a new question, re-rank the similar questions retrieved by a search engine with respect to that question. 1http://alt.qcri.org/semeval2016/task3/ 2 Related Work We build our preprocessing and feature extraction pipeline based on the system of Zamanov et al. (2015), which was developed by a subset of our 2016 team for SemEval-2015 Task 3 on Answer Se- lection in Community Question Answering (Nakov et al., 2015). The task in 2015 was to classify com- ments in a thread as relevant, potentially useful, or bad with respect to the thread question. This year’s Community Question Answering subtask A is sim- ilar to subtask A in 2015, but now it is a ranking task, asking to rank the answers in a thread based on their relevance with respect to the thread’s question. Given this similarity, most of the techniques used by participants in the 2015 subtask A are potentially valuable for this year’s subtask A as well. Below we mention just the few most relevant among them. In their 2015 system, Belinkov et al. (2015) used vectors of the question and of the comment, meta- data features, and text-based similarities. Nicosia et al. (2015) used similarity measures, URLs in the comment text and statistics about the user profile: number of good, bad, and potentially useful com- ments. Similarly, we use the number …

Content truncated to 500 words