SemanticZ at SemEval-2016 Task 3: Ranking Relevant Answers in Community Question Answering Using Semantic Similarity Based on Fine-tuned Word Embeddings

Proceedings of SemEval-2016, pages 879–886, San Diego, California, June 16-17, 2016. c⃝2016 Association for Computational Linguistics SemanticZ at SemEval-2016 Task 3: Ranking Relevant Answers in Community Question Answering Using Semantic Similarity Based on Fine-tuned Word Embeddings Todor Mihaylov Research Training Group AIPHES Institute for Computational Linguistics Heidelberg University mihaylov@cl.uni-heidelberg.de Preslav Nakov Qatar Computing Research Institute, HBKU P.O. box 5825 Doha, Qatar pnakov@qf.org.qa Abstract We describe our system for finding good an- swers in a community forum, as defined in SemEval-2016, Task 3 on Community Ques- tion Answering. Our approach relies on sev- eral semantic similarity features based on fine- tuned word embeddings and topics similar- ities. In the main Subtask C, our primary submission was ranked third, with a MAP of 51.68 and accuracy of 69.94. In Subtask A, our primary submission was also third, with MAP of 77.58 and accuracy of 73.39. 1 Introduction Posting questions that have already been asked and answered in a community forum is annoying to users as it usually ends up with them being referred to a previously asked question. The SemEval-2016 Task 3 on Community Question Answering1(Nakov et al., 2016) aims to solve this real-life problem. The main subtask (Subtask C) asks to find an answer that al- ready exists in the forum and will be appropriate as a response to a newly-posted question. There is also a secondary, Subtask A, which focuses on Question- Comment Similarity and asks to rank the comments within a question-comment thread based on their rel- evance with respect to the thread’s question. Here, we examine the performance of us- ing different word embeddings obtained with the Word2Vec tool (Mikolov et al., 2013), which we use to build vectors for the questions and the answers. We train classifiers using features derived from these embeddings to solve subtasks A and C. 1http://alt.qcri.org/semeval2016/task3/ Our contribution is in producing good word em- beddings based on empirical evaluation of different configurations working in the Community Question Answering domain; as they perform well, we make them freely available to the research community.2 2 Related Work This year’s SemEval-2016 Task 3 is a follow up of SemEval-2015 Task 3 on Answer Selection in Com- munity Question Answering (Nakov et al., 2015). The 2015 subtask A asked to determine whether an answer was relevant, potentially useful, or bad, while this year this is about ranking. Here we focus on features that use semantic knowledge such as word embeddings, various fea- tures extracted from word embeddings, and topic models. Word embeddings and word embeddings similarities have been used by teams in the 2015 edi- tion of the task (Belinkov et al., 2015; Zamanov et al., 2015; Tran et al., 2015; Nicosia et al., 2015). LDA topic have also been used (Tran et al., 2015). Many other features have been tried for the task. For example, Tran et al. (2015) used metadata about the question and the comment. User profile statis- tics such as number of Good, Bad and Potentially Useful comments by a given user have …

Content truncated to 500 words