JU_CSE_NLP: Multi-grade Classification of Semantic Similarity between Text Pairs

First Joint Conference on Lexical and Computational Semantics (*SEM), pages 571–574, Montr´eal, Canada, June 7-8, 2012. c⃝2012 Association for Computational Linguistics JU_CSE_NLP: Multi-grade Classification of Semantic Similarity Between Text Pairs Snehasis Neogi1, Partha Pakray2, Sivaji Bandyopadhyay1 Alexander Gelbukh 1Computer Science & Engineering Department Center for Computing Research Jadavpur University, Kolkata, India National Polytechnic Institute 2Computer Science & Engineering Department Mexico City, Mexico Jadavpur University, Kolkata, India gelbukh@gelbukh.com Intern at Xerox Research Centre Europe Grenoble, France {snehasis1981,parthapakray}@gmail.com sbandyopadhyay@cse.jdvu.ac.in Abstract This article presents the experiments car- ried out at Jadavpur University as part of the participation in Semantic Textual Si- milarity (STS) of Task 6 @ Semantic Evaluation Exercises (SemEval-2012). Task-6 of SemEval- 2012 focused on se- mantic relations of text pair. Task-6 pro- vides five different text pair files to compare different semantic relations and judge these relations through a similarity and confidence score. Similarity score is one kind of multi way classification in the form of grade between 0 to 5. We have submitted one run for the STS task. Our system has two basic modules - one deals with lexical relations and another deals with dependency based syntactic relations of the text pair. Similarity score given to a pair is the average of the scores of the above-mentioned modules. The scores from each module are identified using rule based techniques. The Pearson Correlation of our system in the task is 0.3880. 1 Introduction Task-61 [1] of SemEval-2012 deals with seman- tic similarity of text pairs. The task is to find the similarity between the sentences in the text pair (s1 and s2) and return a similarity score and an optional confidence score. There are five datasets 1 http://www.cs.york.ac.uk/semeval-2012/task6/ in the test data and with tab separated text pairs. The datasets are as follows:  MSR-Paraphrase, Microsoft Research Pa- raphrase Corpus (750 pairs of sentences.)  MSR-Video, Microsoft Research Video De- scription Corpus (750 pairs of sentences.)  SMTeuroparl: WMT2008 development data- set (Europarl section) (459 pairs of sen- tences.)  SMTnews: news conversation sentence pairs from WMT.(399 pairs of sentences.)  OnWN: pairs of sentences where the first comes from Ontonotes and the second from a WordNet definition. (750 pairs of sentences.) Similarity score ranges from 0 to 5 and confi- dence score from 0 to 100. An s1-s2 pair gets a similarity score of 5 if they are completely equivalent. Similarity score 4 is allocated for mostly equivalent s1-s2 pair. Similarly, score 3 is allocated for roughly equivalent pair. Score 2, 1 and 0 are allocated for non-equivalent details sharing, non-equivalent topic sharing and totally different pairs respectively. Major challenge of this task is to find the similarity score based simi- larity for the text pair. Generally text entailment tasks refer whether sentence pairs are entailed or not: binary classification (YES, NO) [2] or multi- classification (Forward, Backward, bidirectional or no entailment) [3][4]. But multi grade classifi- cation of semantic similarity assigns a score to the sentence pair. Our system considers lexical and dependency based syntactic measures for …

Content truncated to 500 words