UniMelb at SemEval-2016 Task 3: Identifying Similar Questions by combining a CNN with String Similarity Measures

Proceedings of SemEval-2016, pages 851–856, San Diego, California, June 16-17, 2016. c⃝2016 Association for Computational Linguistics UniMelb at SemEval-2016 Task 3: Identifying Similar Questions by Combining a CNN with String Similarity Measures Doris Hoogeveen1,2 dhoogeveen@student.unimelb.edu.au Huizhi Liang2 oklianghuizi@gmail.com Long Duong1,2 lduong@student.unimelb.edu.au Yitong Li2 yitongl4@student.unimelb.edu.au Bahar Salehi1,2 bsalehi@student.unimelb.edu.au Timothy Baldwin2 tb@ldwin.net 1NICTA 2Department of Computing and Information Systems The University of Melbourne VIC, Australia Abstract This paper describes the results of the partici- pation of The University of Melbourne in the community question-answering (CQA) task of SemEval 2016 (Task 3-B). We obtained a MAP score of 70.2% on the test set, by com- bining three classifiers: a NaiveBayes clas- sifier and a support vector machine (SVM) each trained over lexical similarity features, and a convolutional neural network (CNN). The CNN uses word embeddings and machine translation evaluation scores as features. 1 Introduction In this paper we present the system we submitted for the community question-answering (CQA) task of the SemEval 2016 workshop (Task 3-B: Nakov et al. (2016)). By finding an automatic way to answer new questions based on existing ones, we unlock an enor- mous wealth of information stored in online CQA archives. In the task as specified for the SemEval workshop, we were given 70 query questions. Each question had at most ten candidate questions, which we were to re-rank according to their similarity to the query question. Each question consisted of a title and a description. The data was taken from the Qatar Living forum.1 The training data set was small: 267 queries, with ten candidate duplicate questions each. The candidate questions were originally labelled ac- cording to the three classes: RELEVANT, PERFECT- MATCH and IRRELEVANT. Subsequently, however, RELEVANT and PERFECTMATCH were merged into a single class. In an ideal ranking, documents with 1http://www.qatarliving.com/forum relevant labels were to be ranked higher than the IR- RELEVANT documents. The system we submitted combines the predic- tions of three different classifiers through simple voting. The first two classifiers (naive Bayes and SVM) made use of semantic similarity measures as features, and the third one was a convolutional neu- ral network (CNN) that used word embeddings and machine translation evaluation scores as input. The combined system achieved a MAP score of 70.2% on the test set. 2 Approach Our system combined the scores of three different classifiers based on simple voting. If at least two of the three classifiers considered a candidate question relevant to a query question, it was considered to be relevant. The candidate questions were then ranked according to this judgement, with the relevant ones on top (in any order, since this was not taken into account in the official evaluation). In this section we will describe the details of the three classifiers. 2.1 String Similarity Features (SS1) Our first set of features consisted of string similar- ity measures, which we selected based on our recent success in applying these features to measure the compositionality of multiword expressions (Salehi and Cook, 2013), to estimate semantic textual simi- larity (Gella …

Content truncated to 500 words