SRIUBC: Simple Similarity Features for Semantic Textual Similarity

First Joint Conference on Lexical and Computational Semantics (*SEM), pages 617–623, Montr´eal, Canada, June 7-8, 2012. c⃝2012 Association for Computational Linguistics SRIUBC: Simple Similarity Features for Semantic Textual Similarity Eric Yeh SRI International Menlo Park, CA USA yeh@ai.sri.com Eneko Agirre University of the Basque Country Donostia, Basque Country e.agirre@ehu.es Abstract We describe the systems submitted by SRI In- ternational and the University of the Basque Country for the Semantic Textual Similarity (STS) SemEval-2012 task. Our systems fo- cused on using a simple set of features, fea- turing a mix of semantic similarity resources, lexical match heuristics, and part of speech (POS) information. We also incorporate pre- cision focused scores over lexical and POS in- formation derived from the BLEU measure, and lexical and POS features computed over split-bigrams from the ROUGE-S measure. These were used to train support vector re- gressors over the pairs in the training data. From the three systems we submitted, two per- formed well in the overall ranking, with split- bigrams improving performance over pairs drawn from the MSR Research Video De- scription Corpus. Our third system maintained three separate regressors, each trained specif- ically for the STS dataset they were drawn from. It used a multinomial classifier to pre- dict which dataset regressor would be most ap- propriate to score a given pair, and used it to score that pair. This system underperformed, primarily due to errors in the dataset predictor. 1 Introduction Previous semantic similarity tasks, such as para- phrase identification or recognizing textual entail- ment, have focused on performing binary decisions. These problems are usually framed in terms of iden- tifying whether a pair of texts exhibit the needed similarity or entailment relationship or not. In many cases, such as producing a ranking over similarity scores, a soft measure of similarity between a pair of texts would be more desirable. We contributed three systems for the 2012 Se- mantic Textual Similarity (STS) task (Agirre et al., 2012). These are: 1. System 1, which used a combination of seman- tic similarity, lexical similarity, and precision focused part-of-speech (POS) features. 2. System 2, which used features from System 1, with the addition of skip-bigram features derived from the ROUGE-S (Lin, 2004) mea- sure. POS variants of skip-bigrams were incor- porated as well. 3. System 3, used the features from above to first classify the dataset the pair was drawn from, and then applied regressors trained for that dataset. Our systems characterize sentence pairs as feature vectors, populated by a variety of scorers that will be described below. During training, we used support vector regression (SVR) to train regressors against these vectors and their associated similarity scores. The STS training data is divided into three datasets, reflecting their origin: Microsoft Research Paraphrase Corpus (MSRpar), MSR Research Video Description Corpus (MSRvid), and WMT2008 De- velopment dataset (SMTeuroparl). We trained indi- vidual regressors for each of these datasets, and ap- plied them to their counterparts in the testing set. Both Systems 1 and 2 used the following types …

Content truncated to 500 words