UOW: Semantically Informed Text Similarity

First Joint Conference on Lexical and Computational Semantics (*SEM), pages 673–678, Montr´eal, Canada, June 7-8, 2012. c⃝2012 Association for Computational Linguistics UOW: Semantically Informed Text Similarity Miguel Rios and Wilker Aziz Research Group in Computational Linguistics University of Wolverhampton Stafford Street, Wolverhampton, WV1 1SB, UK {M.Rios, W.Aziz}@wlv.ac.uk Lucia Specia Department of Computer Science University of Sheffield Regent Court, 211 Portobello, Sheffield, S1 4DP, UK L.Specia@sheffield.ac.uk Abstract The UOW submissions to the Semantic Tex- tual Similarity task at SemEval-2012 use a supervised machine learning algorithm along with features based on lexical, syntactic and semantic similarity metrics to predict the se- mantic equivalence between a pair of sen- tences. The lexical metrics are based on word- overlap. A shallow syntactic metric is based on the overlap of base-phrase labels. The semantically informed metrics are based on the preservation of named entities and on the alignment of verb predicates and the overlap of argument roles using inexact matching. Our submissions outperformed the official base- line, with our best system ranked above aver- age, but the contribution of the semantic met- rics was not conclusive. 1 Introduction We describe the UOW submissions to the Semantic Textual Similarity (STS) task at SemEval-2012. Our systems are based on combining similarity scores as features using a regression algorithm to predict the degree of semantic equivalence between a pair of sentences. We train the regression algorithm with different classes of similarity metrics: i) lexical, ii) syntactic and iii) semantic. The lexical similar- ity metrics are: i) cosine similarity using a bag-of- words representation, and ii) precision, recall and F-measure of content words. The syntactic metric computes BLEU (Papineni et al., 2002), a machine translation evaluation metric, over a labels of base- phrases (chunks). Two semantic metrics are used: a metric based on the preservation of Named Entities and TINE (Rios et al., 2011). Named entities are matched by type and content: while the type has to match exactly, the content is compared with the as- sistance of a distributional thesaurus. TINE is a met- ric proposed to measure adequacy in machine trans- lation and favors similar semantic frames. TINE attempts to align verb predicates, assuming a one- to-one correspondence between semantic roles, and considering ontologies for inexact alignment. The surface realization of the arguments is compared us- ing a distributional thesaurus and the cosine similar- ity metric. Finally, we use METEOR (Denkowski and Lavie, 2010), also a common metric for ma- chine translation evaluation, that also computes in- exact word overlap as at way of measuring the im- pact of our semantic metrics. The lexical and syntactic metrics complement the semantic metrics in dealing with the phenomena ob- served in the task’s dataset. For instance, from the MSRvid dataset: S1 Two men are playing football. S2 Two men are practicing football. In this case, as typical of paraphrasing, the situa- tion and participants are the same while the surface realization differs, but playing can be considered similar to practicing. From the SMT-eur dataset: S3 The Council of Europe, …

Content truncated to 500 words