NLM_NIH at SemEval-2017 Task 3: from Question Entailment to Question Similarity for Community Question Answering

Proceedings of the 11th International Workshop on Semantic Evaluations (SemEval-2017), pages 349–352, Vancouver, Canada, August 3 - 4, 2017. c⃝2017 Association for Computational Linguistics NLM NIH at SemEval-2017 Task 3: from Question Entailment to Question Similarity for Community Question Answering Asma Ben Abacha U.S. National Library of Medicine, Bethesda, MD asma.benabacha@nih.gov Dina Demner-Fushman U.S. National Library of Medicine, Bethesda, MD ddemner@mail.nih.gov Abstract This paper describes our participation in SemEval-2017 Task 3 on Community Question Answering (cQA). The Question Similarity subtask (B) aims to rank a set of related questions retrieved by a search engine according to their similarity to the original question. We adapted our feature- based system for Recognizing Question Entailment (RQE) to the question simi- larity task. Tested on cQA-B-2016 test data, our RQE system outperformed the best system of the 2016 challenge in all measures with 77.47 MAP and 80.57 Ac- curacy. On cQA-B-2017 test data, perfor- mances of all systems dropped by around 30 points. Our primary system obtained 44.62 MAP, 67.27 Accuracy and 47.25 F1 score. The cQA-B-2017 best system achieved 47.22 MAP and 42.37 F1 score. Our system is ranked sixth in terms of MAP and third in terms of F1 out of 13 participating teams. 1 Introduction SemEval-2017 Task 31 on Community Question Answering (cQA) focuses on answering new ques- tions by retrieving related answered questions in community forums (Nakov et al., 2017). This task extends the previous SemEval-2015 and SemEval- 2016 cQA tasks. This year, five subtasks were proposed: En- glish Question-Comment Similarity (subtask A), English Question-Question Similarity (subtask B), English Question-External Comment Similarity (subtask C), Arabic Answer Re-rank (subtask D) and English Multi-Domain Duplicate Question Detection (subtask E). 1http://alt.qcri.org/semeval2017/task3 Subtask B (Question Similarity) aims to re-rank a set of similar questions retrieved by a search en- gine with respect to the original question, with the idea that the answers to the similar questions should also be answers to the new question. For a given question, a set of ten similar questions is provided for re-ranking. 2 Data The cQA task covers two languages: English and Arabic. The English dataset (CQA-QL corpus) is based on data from the Qatar Living forum. The CQA-QL corpus consists of a list of original questions, having each ten related questions from Qatar Living, and the first ten comments from their threads. For subtask B, questions are an- notated as PerfectMatch, Relevant and Irrelevant with respect to the original question. Both Per- fectMatch and Relevant questions are considered as good without distinction. For the cQA-B-2017 task, training and develop- ment datasets are the cQA-B-2016 datasets. A to- tal of 3,869 question pairs is available for training including cQA-2016 test questions. cQA-B-2017 test data is composed of 880 question pairs. 3 Question Similarity vs. Question Entailment In addition to the efforts within the semEval cQA tasks since 2015, earlier definitions and methods were proposed for Question Similarity based on different elements such as the question topic and question type (Burke et al., 1997; Jeon et al., 2005; Duan …

Content truncated to 500 words