SOFTCARDINALITY-CORE: Improving Text Overlap with Distributional Measures for Semantic Textual Similarity

Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 1: Proceedings of the Main Conference and the Shared Task, pages 194–201, Atlanta, Georgia, June 13-14, 2013. c⃝2013 Association for Computational Linguistics SOFTCARDINALITY-CORE: Improving Text Overlap with Distributional Measures for Semantic Textual Similarity Sergio Jimenez, Claudia Becerra Universidad Nacional de Colombia Ciudad Universitaria, edificio 453, oficina 114 Bogotá, Colombia sgjimenezv@unal.edu.co cjbecerrac@unal.edu.co Alexander Gelbukh CIC-IPN Av. Juan Dios Bátiz, esq. Av. Mendizábal, Col. Nueva Industrial Vallejo, CP 07738, DF, México www.gelbukh.com Abstract Soft cardinality has been shown to be a very strong text-overlapping baseline for the task of measuring semantic textual similarity (STS), obtaining 3rd place in SemEval-2012. At *SEM-2013 shared task, beside the plain text- overlapping approach, we tested within soft cardinality two distributional word-similarity functions derived from the ukWack corpus. Unfortunately, we combined these measures with other features using regression, obtain- ing positions 18th, 22nd and 23rd among the 90 participants systems in the official rank- ing. Already after the release of the gold stan- dard annotations of the test data, we observed that using only the similarity measures with- out combining them with other features would have obtained positions 6th, 7th and 8th; more- over, an arithmetic average of these similarity measures would have been 4th(mean=0.5747). This paper describes both the 3 systems as they were submitted and the similarity mea- sures that would obtained those better results. 1 Introduction The task of textual semantic similarity (STS) con- sists in providing a similarity function on pairs of texts that correlates with human judgments. Such a function has many practical applications in NLP tasks (e.g. summarization, question answering, tex- tual entailment, paraphrasing, machine translation evaluation, among others), which makes this task particularly important. Numerous efforts have been devoted to this task (Lee et al., 2005; Mihalcea et al., 2006) and major evaluation campaigns have been held at SemEval-2012 (Agirre et al., 2012) and in *SEM-2013 (Agirre et al., 2013). The experimental setup of STS in 2012 consisted of three data sets, roughly divided in 50% for train- ing and for testing, which contained text pairs manu- ally annotated as a gold standard. Furthermore, two data sets were provided for surprise testing. The measure of performance was the average of the cor- relations per data set weighted by the number of pairs in each data set (mean). The best performing systems were UKP (Bär et al., 2012) mean=0.6773, TakeLab (Šaric et al., 2012) mean=0.6753 and soft cardinality (Jimenez et al., 2012) mean=0.6708. UKP and TakeLab systems used a large number of resources (see (Agirre et al., 2012)) such as dictio- naries, a distributional thesaurus, monolingual cor- pora, Wikipedia, WordNet, distributional similar- ity measures, KB similarity, POS tagger, machine learning and others. Unlike those systems, the soft cardinality approach used mainly text overlapping and conventional text preprocessing such as remov- ing of stop words, stemming and idf term weighting. This shows that the additional gain in performance from using external resources is small and that the soft cardinality approach is a …

Content truncated to 500 words