EICA Team at SemEval-2017 Task 3: Semantic and Metadata-based Features for Community Question Answering

Proceedings of the 11th International Workshop on Semantic Evaluations (SemEval-2017), pages 292–298, Vancouver, Canada, August 3 - 4, 2017. c⃝2017 Association for Computational Linguistics EICA Team at SemEval-2017 Task 3: Semantic and Metadata-based Features for Community Question Answering Yufei Xie, Maoquan Wang, Jing Ma, Jian Jiang, Zhao Lu Department of Computer Science and Technology, East China Normal University, Shanghai, P.R.China yufeixie@ica.stc.sh.cn, zlu@cs.ecnu.edu.cn Abstract We describe our system for participating in SemEval-2017 Task 3 on Community Question Answering. Our approach relies on combining a rich set of various types of features: semantic and metadata. The most important types turned out to be the metadata feature and the semantic vectors trained on QatarLiving data. In the main Subtask C, our primary submission was ranked fourth, with a MAP of 13.48 and accuracy of 97.08. In Subtask A, our pri- mary submission get into the top 50%. 1 Introduction SemEval-2017 Task 3 on Community Question Answering (Nakov et al., 2017) aims to solve a real-life application problem. The main subtask C (Question-External Comment Similarity) asks to find an answer in the forum that is appropriate as a response to a newly posted question. This is achieved by retrieving similar questions and rank- ing their answers with respect to the new question. Three additional supporting subtasks are defined: Subtask A (Question-Comment Similarity): Given a question from a question-comment thread, rank the comments within the thread based on their relevance with respect to the question. The comments in a question-comment thread are anno- tated as Good, PotentiallyUseful and Bad. A good ranking is the one that ranks all Good comments above PotentiallyUseful and Bad ones. Subtask B (Question-Question Similarity): Given a new question, re-rank the similar ques- tions retrieved by a search engine with respect to that question. The potentially relevant question- s are annotated as PerfectMatch, Relevant and Ir- relevant with respect to the original question. A good ranking is the one that the PerfectMatch and the Relevant questions are both ranked above the Irrelevant ones. Subtask C (Question-External Comment Similarity): Given a new question and the set of the first 10 related questions (retrieved by a search engine), each associated with its first 10 comments appearing in its thread. Re-rank the 100 comments (10 questions × 10 comments) according to their relevance with respect to the original question. 2 Related Work This year’s SemEval-2017 Task3 is a follow up of SemEval-2016 Task3 (Nakov et al., 2016) on An- swer Reranking in Community Question Answer- ing. There are three reranking subtasks associated with the English dataset. Subtask A is the same as subtask A at SemEval-2015 Task 3 (Joty et al., 2015), but with slightly different annotation and a different evaluation measure. The research of rerank can be classified into t- wo categories, traditional feature engineering and newest deep neural network employing. The first type of method pays more attention on textural features exploiting. Textual features have been exploited well, including lexical features (e.g., n- grams), syntactic features (such as parse trees) and semantic …

Content truncated to 500 words