bunji at SemEval-2017 Task 3: Combination of Neural Similarity Features and Comment Plausibility Features

Proceedings of the 11th International Workshop on Semantic Evaluations (SemEval-2017), pages 353–359, Vancouver, Canada, August 3 - 4, 2017. c⃝2017 Association for Computational Linguistics bunji at SemEval-2017 Task 3: Combination of Neural Similarity Features and Comment Plausibility Features Yuta Koreeda1, Takuya Hashito2, Yoshiki Niwa1, Misa Sato1, Toshihiko Yanase1, Kenzo Kurotsuchi1, and Kohsuke Yanai1 1Research & Development Group, Hitachi, Ltd. 2Industry & Distribution Business Unit, Hitachi, Ltd. {yuta.koreeda.pb, takuya.hashito.qk, yoshiki.niwa.tx, misa.sato.mw toshihiko.yanase.gm, kenzo.kurotsuchi.qs, kohsuke.yanai.cs}@hitachi.com Abstract This paper describes a text-ranking system developed by bunji team in SemEval-2017 Task 3: Community Question Answering, Subtask A and C. The goal of the task is to re-rank the comments in a question-and- answer forum such that useful comments for answering the question are ranked high. We proposed a method that com- bines neural similarity features and hand- crafted comment plausibility features, and we modeled inter-comments relationship using conditional random field. Our ap- proach obtained the fifth place in the Sub- task A and the second place in the Subtask C. 1 Introduction This paper explains the participation of the bunji team in SemEval-2017 Task 3 on Community Question Answering (CQA) (Nakov et al., 2017), Subtask A and Subtask C. The goal of the task is to re-rank the comments in a question-and-answer forum such that useful comments for answering the question are ranked high. Subtask A is extrac- tion of relevant answers from comments in a ques- tion thread. Given a question and its comments, the system must re-rank the comments accord- ing to their relevance with respect to the question. Subtask C is extraction of relevant answers from comments in different question threads. Given a question (the original question), questions that are possibly related to the original question (the relevant questions) and comments to the relevant questions, the system must re-rank the comments according to their relevance with respect to the original question. Since the task is ranking, the primary metric is mean average precision (MAP). Our model consists of three elements; use of similarity features, use of comment plausibility features and a supervised scoring method that models inter-comments relationship. The similar- ity features are designed to capture the similarities between a question and a comment because a valid answer should be on the same topic as the ques- tion. Similarity features were utilized by many teams in SemEval-2016 (Nakov et al., 2016). In this work, we take a deep learning approach to ex- tract similarity features. The comment plausibility features are designed to capture characteristics that relevant answers tend to have. Similar concept was proposed by Mihaylova et al. (2016), who tried to model read- ability, credibility, sentiment and trollness. The comment plausibility features were hand-crafted to incorporate human knowledge about CQA. In past CQA tasks, some teams incorporated inter-comments relationship. An example of such relationship is acknowledgement, where a good answer is likely to be followed by acknowledge- ment of the questioner. Barrn-Cedeo et al. (2015) modeled inter-comments relationship by taking distance to nearest acknowledgement as a feature and using Conditional …

Content truncated to 500 words