FBK-HLT: An Effective System for Paraphrase Identification and Semantic Similarity in Twitter

Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 29–33, Denver, Colorado, June 4-5, 2015. c⃝2015 Association for Computational Linguistics FBK-HLT: An Effective System for Paraphrase Identification and Semantic Similarity in Twitter Ngoc Phuoc An Vo University of Trento, Fondazione Bruno Kessler Trento, Italy ngoc@fbk.eu Simone Magnolini University of Brescia, Fondazione Bruno Kessler Trento, Italy magnolini@fbk.eu Octavian Popescu IBM Research, T.J. Watson Yorktown, US o.popescu@us.ibm.com Abstract This paper reports the description and perfor- mance of our system, FBK-HLT, participating in the SemEval 2015, Task #1 "Paraphrase and Semantic Similarity in Twitter", for both sub- tasks. We submitted two runs with different classifiers in combining typical features (lexi- cal similarity, string similarity, word n-grams, etc) with machine translation metrics and edit distance features. We outperform the baseline system and achieve a very competitive result to the best system on the first subtask. Eventually, we are ranked 4th out of 18 teams participating in subtask "Paraphrase Identification". 1 Introduction Paraphrase identification/recognition is an important task that can be used as a feature to improve many other NLP tasks as Information Retrieval, Machine Translation Evaluation, Text Summarization, Ques- tion and Answering, and others. Besides this, analyz- ing social data like tweets of social network Twitter is a field of growing interest for different purposes. The interesting combination of these two tasks was brought forward as Shared Task #1 in the SemEval 2015 campaign for "Paraphrase and Semantic Simi- larity in Twitter" (Xu et al., 2015). In this task, given a set of sentence pairs, which are not necessarily full tweets, their topic and the same sentences with part- of-speech and named entity tags; participating sys- tem is required to predict for each pair of sentences is a paraphrase (Subtask 1) and optionally compute a graded score between 0 and 1 for their semantic equivalence (Subtask 2). We participate in this shared task with a system combining different features us- ing a binary classifier. We are interested in finding out whether semantic similarity, textual entailment and machine translation evaluation techniques could increase the accuracy of our system. This paper is organized as follows: Section 2 presents the System Description, Section 3 describes the Experiment Set- tings, Section 4 reports the Evaluations, Section 5 shows the Error Analysis, and finally Section 6 is the Conclusions and Future Work. 2 System Description In order to build our system, we extract and select sev- eral different linguistic features ranging from simple (word/string similarity, edit distance) to more com- plex ones (machine translation evaluation metrics), then we consolidate them by a binary classifier. More- over, different features can be used independently or together with others to measure the semantic similar- ity and recognize the paraphrase of given sentence pair as well as to evaluate the significance of each feature to the accuracy of system’s predictions. On top of this, the system is expandable and scalable for adopting more useful features aiming for improving the accuracy. 2.1 Data Preprocessing In order to optimizing the system performance, …

Content truncated to 500 words