TakeLab: Systems for Measuring Semantic Text Similarity

First Joint Conference on Lexical and Computational Semantics (*SEM), pages 441–448, Montr´eal, Canada, June 7-8, 2012. c⃝2012 Association for Computational Linguistics TakeLab: Systems for Measuring Semantic Text Similarity Frane ˇSari´c, Goran Glavaˇs, Mladen Karan, Jan ˇSnajder, and Bojana Dalbelo Baˇsi´c University of Zagreb Faculty of Electrical Engineering and Computing {frane.saric, goran.glavas, mladen.karan, jan.snajder, bojana.dalbelo}@fer.hr Abstract This paper describes the two systems for determining the semantic similarity of short texts submitted to the SemEval 2012 Task 6. Most of the research on semantic similarity of textual content focuses on large documents. However, a fair amount of information is con- densed into short text snippets such as social media posts, image captions, and scientific ab- stracts. We predict the human ratings of sen- tence similarity using a support vector regres- sion model with multiple features measuring word-overlap similarity and syntax similarity. Out of 89 systems submitted, our two systems ranked in the top 5, for the three overall eval- uation metrics used (overall Pearson – 2nd and 3rd, normalized Pearson – 1st and 3rd, weighted mean – 2nd and 5th). 1 Introduction Natural language processing tasks such as text clas- sification (Sebastiani, 2002), text summarization (Lin and Hovy, 2003; Aliguliyev, 2009), informa- tion retrieval (Park et al., 2005), and word sense dis- ambiguation (Sch¨utze, 1998) rely on a measure of semantic similarity of textual documents. Research predominantly focused either on the document sim- ilarity (Salton et al., 1975; Maguitman et al., 2005) or the word similarity (Budanitsky and Hirst, 2006; Agirre et al., 2009). Evaluating the similarity of short texts such as sentences or paragraphs (Islam and Inkpen, 2008; Mihalcea et al., 2006; Oliva et al., 2011) received less attention from the research community. The task of recognizing paraphrases (Michel et al., 2011; Socher et al., 2011; Wan et al., 2006) is sufficiently similar to reuse some of the techniques. This paper presents the two systems for auto- mated measuring of semantic similarity of short texts which we submitted to the SemEval-2012 Se- mantic Text Similarity Task (Agirre et al., 2012). We propose several sentence similarity measures built upon knowledge-based and corpus-based similarity of individual words as well as similarity of depen- dency parses. Our two systems, simple and syn- tax, use supervised machine learning, more specif- ically the support vector regression (SVR), to com- bine a large amount of features computed from pairs of sentences. The two systems differ in the set of features they employ. Our systems placed in the top 5 (out of 89 sub- mitted systems) for all three aggregate correlation measures: 2nd (syntax) and 3rd (simple) for overall Pearson, 1st (simple) and 3rd (syntax) for normal- ized Pearson, and 2nd (simple) and 5th (syntax) for weighted mean. The rest of the paper is structured as follows. In Section 2 we describe both knowledge-based and corpus-based word similarity measures. In Section 3 we describe in detail the features used by our sys- tems. In Section 4 we report the experimental results cross-validated on the development set as well …

Content truncated to 500 words