Talla at SemEval-2017 Task 3: Identifying Similar Questions Through Paraphrase Detection

Proceedings of the 11th International Workshop on Semantic Evaluations (SemEval-2017), pages 375–379, Vancouver, Canada, August 3 - 4, 2017. c⃝2017 Association for Computational Linguistics Talla at SemEval-2017 Task 3: Identifying Similar Questions Through Paraphrase Detection Byron V. Galbraith, Bhanu Pratap, Daniel Shank Talla Boston, MA, USA {byron, bhanu, daniel}@talla.com Abstract This paper describes our approach to the SemEval-2017 shared task of determining question-question similarity in a commu- nity question-answering setting (Task 3B). We extracted both syntactic and seman- tic similarity features between candidate questions, performed pairwise-preference learning to optimize for ranking order, and then trained a random forest classifier to predict whether the candidate questions were paraphrases of each other. This ap- proach achieved a MAP of 45.7% out of max achievable 67.0% on the test set. 1 Introduction A large amount of information of interest to users of community forums is stored in semi-structured text, but surfacing that information can be chal- lenging given the variety of ways users can phrase their search queries. Question-answering is a sig- nificant task for both natural language processing (NLP) and information retrieval (IR), as both the actual terms used in the query plus the seman- tic intent of the query itself need to be accounted for in surfacing relevant potential answers. The Community Question Answering (cQA) task of SemEval-2017 (Nakov et al., 2017) seeks to ad- dress this problem through several related sub- tasks around effectively determining and ranking the relevance of related stored questions and asso- ciated answers. We chose to focus on subtask B: question- question similarity. This problem can be seen as one of paraphrase detection – determine if two questions have the same meaning. We reviewed existing performant paraphrase detection meth- ods and selected several to implement and ensem- ble (Ji and Eisenstein, 2013; Wan et al., 2006; Wang and Ittycheriah, 2015; Filice et al., 2015) along with the related question IR system rank provided in the dataset. As paraphrase detection is a classification problem while subtask B is a ranking problem, we also incorporated pairwise- preference learning (Joachims, 2002; F¨urnkranz and H¨ullermeier, 2003) to aid in improving the key metric of mean average precision (MAP). The rest of the paper is organized as follows. Section 2 provides a detailed description of our system, including the key identified features that were extracted, while Section 3 provides the re- sults from experiments used to evaluate the sys- tem. Section 4 concludes the paper with a sum- mary of the work and directions for future explo- ration. 2 System Description Our approach consisted of four parts: data prepa- ration, feature extraction, pairwise-preference learning, and paraphrase classification. All code was implemented in Python 3.5. For data extrac- tion, we converted the XML documents provided by (Nakov et al., 2017) into pandas DataFrames, retaining the subject text, body text, and meta- data related to the original and related questions. The feature extraction and the pairwise-preference learning phase are described below. Classification was handled with a random forest classifier con- taining 2000 weak …

Content truncated to 500 words