FBK: Machine Translation Evaluation and Word Similarity metrics for Semantic Textual Similarity

First Joint Conference on Lexical and Computational Semantics (*SEM), pages 624–630, Montr´eal, Canada, June 7-8, 2012. c⃝2012 Association for Computational Linguistics FBK: Machine Translation Evaluation and Word Similarity metrics for Semantic Textual Similarity Jos´e Guilherme C. de Souza Fondazione Bruno Kessler University of Trento Povo, Trento, Italy desouza@fbk.eu Matteo Negri Fondazione Bruno Kessler Povo, Trento Italy negri@fbk.eu Yashar Mehdad Fondazione Bruno Kessler Povo, Trento Italy mehdad@fbk.eu Abstract This paper describes the participation of FBK in the Semantic Textual Similarity (STS) task organized within Semeval 2012. Our ap- proach explores lexical, syntactic and se- mantic machine translation evaluation metrics combined with distributional and knowledge- based word similarity metrics. Our best model achieves 60.77% correlation with hu- man judgements (Mean score) and ranked 20 out of 88 submitted runs in the Mean rank- ing, where the average correlation across all the sub-portions of the test set is considered. 1 Introduction The Semantic Textual Similarity (STS) task pro- posed at SemEval 2012 consists of examining the degree of semantic equivalence between two sen- tences and assigning a score to quantify such sim- ilarity ranging from 0 (the two texts are about dif- ferent topics) to 5 (the two texts are semantically equivalent). The complete description of the task, the datasets and the evaluation methodology adopted can be found in (Agirre et al., 2012). Typical approaches to measure semantic textual similarity exploit information at the lexical level. The proposed solutions range from calculating the overlap of common words between the two text seg- ments (Salton et al., 1997) to the application of knowledge-based and corpus-based word similarity metrics to cope with the low recall achieved by on simple lexical matching (Mihalcea et al., 2006). Our participation in the STS task is inspired by previous work on paraphrase recognition, in which machine translation (MT) evaluation metrics are used to identify whether a pair of sentences are semantically equivalent or not (Finch and Hwang, 2005; Wan et al., 2006). Our approach to semantic textual similarity makes use of not only lexical in- formation but also syntactic and semantic informa- tion. To this aim, our metrics are based on different natural language processing tools that provide syn- tactic and semantic annotation. These include shal- low parsing, constituency parsing, dependency pars- ing, semantic roles labeling, discourse representa- tion analyzer, and named entities recognition. In ad- dition, we employed distributional and knowledge- based word similarity metrics in an attempt to im- prove the results given by the MT metrics. The com- puted scores are used as features to train a regression model in a supervised learning framework. Our best run model achieves 60.77% correlation with human judgements when evaluating the seman- tic similarity of texts from the entire test set and was ranked in the 20th position (out of 88 submit- ted runs) in the Mean ranking. 2 System Description The system has been designed following a ma- chine learning based approach in which a regres- sion model is induced using different shallow and deep linguistic features extracted from …

Content truncated to 500 words