Overfitting at SemEval-2016 Task 3: Detecting Semantically Similar Questions in Community Question Answering Forums with Word Embeddings

Proceedings of SemEval-2016, pages 861–865, San Diego, California, June 16-17, 2016. c⃝2016 Association for Computational Linguistics Overfitting at SemEval-2016 Task 3: Detecting Semantically Similar Questions in Community Question Answering Forums with Word Embeddings Hujie Wang University of Waterloo 200 University Ave W Waterloo, ON N2L 3G1, Canada h3wang@uwaterloo.ca Pascal Poupart University of Waterloo 200 University Ave W Waterloo, ON N2L 3G1, Canada ppoupart@uwaterloo.ca Abstract This paper presents an approach for estimat- ing the question-question similarity of an En- glish dataset specified in Shared Task 3, sub- task B of SemEval-2016. Given a new ques- tion and a set of the first 10 related questions retrieved by a search engine, participants are asked to produce a binary relevant/irrelevant judgement and rerank the related questions ac- cording to their similarity with respect to the original question. Our submitted system uses a 2-layer feed-forward neural network with the averages of word embedding vectors to predict the semantic similarity score of two questions. We also evaluate the results of Ran- dom Forests and Support Vector Machine in comparison to the Neural Network. Results on the test dataset show that the model achieves a Mean Average Precision 69.681. 1 Introduction Finding semantically related questions is a diffi- cult task due to two main factors: (1) paraphrasing, which can appear at different levels, e.g., lexical, phrasal, sentential (Madnani and Dorr, 2010); and (2) two questions could be asking different things but look for the same solution. Thus, traditional sur- face similarity measures such as Edit Distance, Jac- card, Dice and Overlap coefficient and their varia- tions are not able to capture many cases of semantic relatedness. To preserve the semantic meaning of a question, we use weighted average word embed- ding vectors to model questions. We evaluate sev- eral algorithms that take the weighted average word embedding vectors of two questions as inputs and predict whether two questions are related or not. To provide a benchmark so as to compare and de- velop question-question similarity measuring mod- els in Community Question Answering forums, the Question-Question Similarity task in SemEval-2016 Task 3, Subtask B (Nakov et al., 2016) requires the participants to determine whether two questions are sementically related and given a new question, rerank all similar questions retrieved by a search en- gine. Table 1 presents an example of a pair of se- mantically related questions. We perform an extensive number of experiments using data from Shared Task 3 and compare three types of classifiers: Neural Network (NN), Support Vector Machine (SVM) and Random Forests (RF), as well as the impact of different weighting schemes for averaging word embedding vectors and differ- ent word embedding dimensions. The results show that 2-layer Feedforward Neural Network with Idf weighting scheme and in-domain word embeddings outperforms the rest of models. Title: Best Bank Body: Which is a good bank as per your expe- rience in Doha Title: Good Bank Body: Hi Guys; I need to open a new bank ac- coount. Which is the best bank in Qatar ? …

Content truncated to 500 words