FBK-HLT: An Application of Semantic Textual Similarity for Answer Selection in Community Question Answering

Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 231–235, Denver, Colorado, June 4-5, 2015. c⃝2015 Association for Computational Linguistics FBK-HLT: An Application of Semantic Textual Similarity for Answer Selection in Community Question Answering Ngoc Phuoc An Vo University of Trento, Fondazione Bruno Kessler Trento, Italy ngoc@fbk.eu Simone Magnolini University of Brescia, Fondazione Bruno Kessler Trento, Italy magnolini@fbk.eu Octavian Popescu IBM Research, T.J. Watson Yorktown, US o.popescu@us.ibm.com Abstract This paper reports the description and perfor- mance of our system, FBK-HLT, participating in the SemEval 2015, Task #3 "Answer Se- lection in Community Question Answering" for English, for both subtasks. We submit two runs with different classifiers in combining typ- ical features (lexical similarity, string similar- ity, word n-grams, etc.) with machine transla- tion evaluation metrics and with some ad hoc features (e.g user overlapping, spam filtering). We outperform the baseline system and achieve interesting results on both subtasks. 1 Introduction Answer selection is an important task inside the wider task of question answering that represents at the mo- ment a topic of great interest for research and busi- ness as well. Analyzing social data like answers given inside a forum is a way to maximize the value of this type of knowledge source that is usually af- fected by a very noisy information due to out of topic spam, double posting, cross posting or other issues. Recognizing useful posts from bad ones, and auto- matically detecting the main polarity of answers to a given question is a way to treat an amount of data that otherwise might be difficult to handle. A promising way to provide insight into these ques- tions was brought forward as Shared Task #3 in the SemEval-2015 campaign for "Answer Selection in Community Question Answering" (Màrquez et al., 2015) for English and Arabic languages. In the Sub- task A, each system is given a set of questions in which each one contains some data like posting date, author’s Id, a set of comments, at least one, but usu- ally more; then the participating the system has to classify comments as good, bad or potential accord- ing to their relevance with the question. In Subtask B, a subset of these questions are predefined as yes/no questions, system has to classify them into yes, no or unsure classes based on the individual good answers. We participate in this shared task (only in English) with a system composing several different features using a multiclass classifier. We are interested in finding out whether similarity, machine translation evaluation metrics and task specific techniques could increase the accuracy of our system. In this paper, we outline our method and present the results for the answer selection task; the paper is organized as follows: Section 2 presents the System Description, Section 3 describes the Experiment Settings, Section 4 reports the Evaluations, Section 5 is the Error Anal- ysis and finally, Section 6 presents the Conclusions and Future Work. 2 System Description In order to build our system, we extract and …

Content truncated to 500 words