Beihang-MSRA at SemEval-2017 Task 3: A Ranking System with Neural Matching Features for Community Question Answering

Proceedings of the 11th International Workshop on Semantic Evaluations (SemEval-2017), pages 280–286, Vancouver, Canada, August 3 - 4, 2017. c⃝2017 Association for Computational Linguistics Beihang-MSRA at SemEval-2017 Task 3: A Ranking System with Neural Matching Features for Community Question Answering Wenzheng Feng†, Yu Wu†, Wei Wu‡, Zhoujun Li†∗, Ming Zhou‡ †State Key Lab of Software Development Environment, Beihang University, Beijing, China ‡ Microsoft Research, Beijing, China {wuyu,lizj,wenzhengfeng}@buaa.edu.cn {wuwei,mingzhou}@microsoft.com Abstract This paper presents the system SemEval-2017 Task 3, Community Ques-in tion Answering (CQA). We develop a ranking system that is capable of captur- ing semantic relations between text pairs with little word overlap. In addition to traditional NLP features, we introduce several neural network based matching features which enable our system to mea- sure text similarity beyond lexicons. Our system significantly outperforms baseline methods and holds the second place in Subtask A and the fifth place in Subtask B, which demonstrates its efficacy on answer selection and question retrieval. 1 Introduction In task 3 of SemEval 2017, participants are required to address typical problems in mod- ern CQA forums. We participate two sub- tasks: question-comment similarity (Subtask A) and question-question similarity (Subtask B). In Subtask A, given a question and 10 comments in its comment thread, one is required to re-rank the 10 comments according to their relevance with the question. Subtask B gives a question and asks par- ticipants to re-rank 10 related questions according to their similarity to the input question. The challenge of both subtasks is that two natu- ral language sentences often express similar mean- ings with different but semantically related words, which results in semantic gaps between them. To bridge the semantic gaps, we build a ranking sys- tem with a variety of features. In addition to tra- ditional NLP features such as tf-idf (Salton and Buckley, 1988), the longest common subsequence (Allison and Dix, 1986), translation models (Jeon ∗ Corresponding Author et al., 2005), and tree kernels (Schlkopf et al., 2003; Collins and Duffy, 2002; Moschitti, 2006), which match sentences based on word overlap, syntax (tree kenerls), and word-word translations (translation models), we also introduce neural net- work based matching models into the system as features. The neural matching features, includ- ing a long short term memory network (LSTM) (Schuster and Paliwal, 1997) and a 2D matching network which is a variant of our model in (Wu et al., 2016), can extract high level matching sig- nals from distributed representations of the sen- tences and capture their similarity beyond lexi- cons. We also design some specific features for each subtask. All the features are combined as a ranking model by a gradient boosted regression tree which is implemented by Xgboost (Chen and Guestrin, 2016). Our system significantly outper- forms baseline methods on the two subtasks. On Subtask A, it holds the second place and is compa- rable with the best system. On Subtask B, it holds the fifth place. The results demonstrate that our system can alleviate the semantic gaps in the tasks of CQA …

Content truncated to 500 words