UoB-UK at SemEval-2016 Task 1: A Flexible and Extendable System for Semantic Text Similarity using Types, Surprise and Phrase Linking

Proceedings of SemEval-2016, pages 680–685, San Diego, California, June 16-17, 2016. c⃝2016 Association for Computational Linguistics UoB-UK at SemEval-2016 Task 1: A Flexible and Extendable System for Semantic Text Similarity using Types, Surprise and Phrase Linking Harish Tayyar Madabushi, Mark Buhagiar, Mark Lee School of Computer Science, University of Birmingham, Birmingham, United Kingdom. H.T.Madabushi@cs.bham.ac.uk, Buhagiar.Mark.A@gmail.com, M.G.Lee@cs.bham.ac.uk Abstract We present in this paper a system for measur- ing Semantic Text Similarity (STS) in English. We introduce three novel techniques: the use of Types, methods of linking phrases, and the use of a Surprise Factor to generate 8,370 sim- ilarity measures, which we then combine us- ing Support Vector and Kernel Ridge Regres- sion. Our system out performs the State of the Art in SemEval 2015, and our best performing run achieved a score of .7094 on the 2016 test set as a whole, and over 0.8 on the majority of the datasets. Additionally, the use of Sur- prise, Types and phrase linking is not limited to STS and can be used across various Natural Language Processing tasks, while our method of combining scores provides a flexible way of combining variously generated Similarity Scores. 1 Introduction and Motivation The goal of Semantic Text Similarity (STS) is to find the degree of overlap in the meaning of two pieces of text. This ranges from text fragments that are exact semantic equivalents, to others that have no semantic relation. STS has a wide variety of applications, in- cluding text summarisation (Aliguliyev, 2009), ma- chine translation (Kauchak and Barzilay, 2006), and search optimisation (Sriram et al., 2010). The STS task, which has been set by the SemEval conference for the past number of years (Agirre et al., 2014; Agirre et al., 2015), requires that submit- ted systems assign a score between 0 (the sentences are on different topics) and 5 (the sentences mean exactly the same thing) that reflects how similar two sentences are. Most systems that tackled SemEval’s STS task in previous years have involved three main approaches: The first is text alignment, based on the content words’ meaning (Sultan et al., 2015; Sultan et al., 2014b). The second represents text as vectors, which are used to find the similarity score using a vector similarity metric (such as cosine). Third, ma- chine learning approaches are used to compute mul- tiple lexical, semantic, and syntactic features to clas- sify each sentence pair’s similarity. We make use of both text alignments and vec- tor representations, while limiting comparisons to words of the same Type (Section 4.1), a novel con- cept we introduce in addition to methods of phrase linking (Section 4.2) and establishing common noun importance (Section 4.3). These, combined with several different weight combinations we pick for each word Type, provide us with 8,370 semantic relation measures (Section 5). The overall algo- rithm for generating the several similarity measures is presented in Algorithm 1. We choose a subset of these measures using methods detailed in Sec- tion 6.1, combine them with a limited set of …

Content truncated to 500 words