TakeLab-QA at SemEval-2017 Task 3: Classification Experiments for Answer Retrieval in Community QA

Proceedings of the 11th International Workshop on Semantic Evaluations (SemEval-2017), pages 339–343, Vancouver, Canada, August 3 - 4, 2017. c⃝2017 Association for Computational Linguistics TakeLab-QA at SemEval-2017 Task 3: Classification Experiments for Answer Retrieval in Community QA Filip Šaina, Toni Kukurin, Lukrecija Pulji´c, Mladen Karan, Jan Šnajder Text Analysis and Knowledge Engineering Lab Faculty of Electrical Engineering and Computing, University of Zagreb Unska 3, 10000 Zagreb, Croatia {name.surname}@fer.hr Abstract In this paper we present the TakeLab-QA entry to SemEval 2017 task 3, which is a question-comment re-ranking problem. We present a classification based approach, in- cluding two supervised learning models – Support Vector Machines (SVM) and Con- volutional Neural Networks (CNN). We use features based on different semantic similarity models (e.g., Latent Dirichlet Allocation), as well as features based on several types of pre-trained word embed- dings. Moreover, we also use some hand- crafted task-specific features. For training, our system uses no external labeled data apart from that provided by the organiz- ers. Our primary submission achieves a MAP-score of 81.14 and F1-score of 66.99 – ranking us 10th on the SemEval 2017 task 3, subtask A. 1 Introduction The ever-growing Community Question Answering (CQA) on-line services are gaining popularity at an increasing rate. However, there are some problems inherent to question-answer collections created by on-line communities. A major issue is the sheer volume of CQA collections, which makes finding an answer to a user question infeasible without some kind of an automated retrieval system. Conse- quently, information retrieval on CQA collections has gained increased focus in the research commu- nity, giving rise to several shared tasks on SemEval (Nakov et al., 2017). From a natural language processing perspective, this is a difficult task due to high variance in the quality of questions and answers in CQA collec- tions (Màrquez et al., 2015). The cause of this is the self-moderated nature of CQA sites, which implies that there are few restrictions on who is allowed to answer a question. In this paper we describe our entries for the Se- mEval 2017 Question-Comment Similarity subtask (Nakov et al., 2017). Given a question q and a comment list C, the task is to rank the comments in C according to their relevance with respect to q. Datasets for this task were extracted from Qatar Living, a web forum where people pose questions about various aspects of their daily life in Qatar. Following Filice et al. (2016), we framed the task as a binary classification problem. We experi- mented with two classification approaches – Sup- port Vector Machines (SVM) (Cortes and Vapnik, 1995) and Convolutional Neural Networks (CNN) (Kim, 2014). Most of the features we use follow the work of Barrón-Cedeno et al. (2015) and Nicosia et al. (2015). Moreover, we use embedding-based (Mihaylov and Nakov, 2016) features for both mod- els. The CNN model with the full feature set has proven to be the most successful, ranking 10th in the competition with a MAP-score of 81.14 and an F1-score of 66.99. The …

Content truncated to 500 words