LearningToQuestion at SemEval 2017 Task 3: Ranking Similar Questions by Learning to Rank Using Rich Features

Proceedings of the 11th International Workshop on Semantic Evaluations (SemEval-2017), pages 310–314, Vancouver, Canada, August 3 - 4, 2017. c⃝2017 Association for Computational Linguistics LearningToQuestion at SemEval 2017 Task 3: Ranking Similar Questions by Learning to Rank Using Rich Features Naman Goyal Georgia Institute of Technology naman.goyal21@gmail.com Abstract This paper describes our official entry LearningToQuestion for SemEval 2017 task 3 community question answer, sub- task B. The objective is to rerank ques- tions obtained in web forum as per their similarity to original question. Our system uses pairwise learning to rank methods on rich set of hand designed and represen- tation learning features. We use various semantic features that help our system to achieve promising results on the task. The system achieved second highest results on official metrics MAP and good results on other search metrics. 1 Introduction In online forums question answering is one of the most popular way for users to share information between each other. Due to the unstructured na- ture of these forums, it’s a problem to find rele- vant information from the already existing infor- mation for users. One way to solve this problem is to design systems to automatically find similar content (question, answer, comment) to the user’s posted question. SemEval-2017 task 3 (Nakov et al., 2017) focuses on solving this problem in community question answer by various subtasks of ranking relevant information in Qatar living fo- rums data. The system presented in this paper fo- cuses on subtask B, to re-rank given set of ques- tions retrieved by search engine, in their similarity to original question. The system is mainly designed by employing learning to rank methods on the rich feature set obtained by text processing of the question text. 2 Data We primarily use the annotated training, develop- ment and testing dataset provided by the SemEval- 2017 task 3 organizers. The dataset is collected by organizers from Qatar living forum. It’s in the form of an original question and set of related questions. Each related question in training and development dataset is annotated with one of the 3 possible tags, PerfectMatch, Relevant or Irrele- vant. A ranking task is required to rank both Per- fectMatch and Relevant above Irrelevant questions without any distinction between the first two. The train dataset for subtask B consists of 317 original questions and 3169 retrieved questions by search engine roughly 10 related questions per original question. The organizers have also provided anno- tated test dataset from SemEval-2016 challenge. Along with these we also used Glove embed- dings (Pennington et al., 2014) which were pre- trained using 6 billion tokens from Wikipedia- 2014 and Gigaword dataset. 3 System Since the task is a ranking task, our system uses learning to rank (Trotman, 2005) to model the ranking of questions. Learning to rank refers to various machine learning techniques used in rank- ing tasks. These have been studied in informa- tion retrieval literature and they power many of the industrial search engines. These systems mainly fall into 3 …

Content truncated to 500 words