FBK-HLT: A New Framework for Semantic Textual Similarity

Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 102–106, Denver, Colorado, June 4-5, 2015. c⃝2015 Association for Computational Linguistics FBK-HLT: A New Framework for Semantic Textual Similarity Ngoc Phuoc An Vo Fondazione Bruno Kessler, University of Trento Trento, Italy ngoc@fbk.eu Simone Magnolini University of Brescia, Fondazione Bruno Kessler Trento, Italy magnolini@fbk.eu Octavian Popescu IBM Research, T.J. Watson Yorktown, US o.popescu@us.ibm.com Abstract This paper reports the description and perfor- mance of our system, FBK-HLT, participat- ing in the SemEval 2015, Task #2 “Semantic Textual Similarity”, English subtask. We sub- mitted three runs with different hypothesis in combining typical features (lexical similarity, string similarity, word n-grams, etc) with syn- tactic structure features, resulting in different sets of features. The results evaluated on both STS 2014 and 2015 datasets prove our hypoth- esis of building a STS system taking into con- sideration of syntactic information. We out- perform the best system on STS 2014 datasets and achieve a very competitive result to the best system on STS 2015 datasets. 1 Introduction Semantic related tasks have been a noticed trend in Natural Language Processing (NLP) community. Particularly, the task Semantic Textual Similarity (STS) has captured a huge attention in the NLP com- munity despite being recently introduced since Se- mEval 2012 (Agirre et al., 2012). Basically, the task requires to build systems which can compute the similarity degree between two given sentences. The similarity degree is scaled as a real score from 0 (no relevance) to 5 (semantic equivalence). The evaluation is done by computing the correlation be- tween human judgment scores and system scores by the mean of Pearson correlation method. At SemEval 2015, Task #2 “Semantic Textual Similarity (STS)”, English STS subtask (Agirre et al., 2015) evaluates participating systems on five test datasets: image description (image), news headlines (headlines), student answers paired with reference answers (answers-students), answers to questions posted in stach exchange forums (answers-forum), and English discussion forum data exhibiting com- mited belief (belief). As being inspired by the UKP system (Bär et al., 2012), which was the best system in STS 2012, we build a supervised system on top of it. Our system adopts some word and string similar- ity features in UKP, such as string similarity, charac- ter/word n-grams, and pairwise similarity; however, we also add other distinguished features, like syn- tactic structure information, word alignment and se- mantic word similarity. As a result, our team, FBK- HLT, submitted three runs and achieve very compet- itive results in the top-tier systems of the task. The remainder of this paper is organized as fol- lows: Section 2 presents the System Description, Section 3 describes our Experiment Settings, Sec- tion 4 reports the Evaluations of our system. Finally, Section 5 is Conclusions and Future Work. 2 System Description We describe our system, which is built from differ- ent linguistic features. We construct a pipeline sys- tem, in which each component produces different features independently and at the end, all features are consolidated by a machine learning tool, …

Content truncated to 500 words