DLS@CU at SemEval-2016 Task 1: Supervised Models of Sentence Similarity

Proceedings of SemEval-2016, pages 650–655, San Diego, California, June 16-17, 2016. c⃝2016 Association for Computational Linguistics DLS@CU at SemEval-2016 Task 1: Supervised Models of Sentence Similarity Md Arafat Sultan† Steven Bethard‡ Tamara Sumner† †Institute of Cognitive Science and Department of Computer Science, University of Colorado Boulder ‡Department of Computer and Information Sciences, University of Alabama at Birmingham, arafat.sultan@colorado.edu, bethard@uab.edu, sumner@colorado.edu Abstract We describe a set of systems submitted to the SemEval-2016 English Semantic Textual Sim- ilarity (STS) task. Given two English sen- tences, the task is to compute the degree of their semantic similarity. Each of our systems uses the SemEval 2012–2015 STS datasets to train a ridge regression model that combines different measures of similarity. Our best sys- tem demonstrates 73.6% correlation with av- erage human annotations across five test sets. 1 Introduction Identification of short-text semantic similarity is an important research problem with application in a multitude of NLP tasks: question answering (Yao et al., 2013; Severyn and Moschitti, 2013), short an- swer grading (Mohler et al., 2011; Ramachandran et al., 2015), text summarization (Dasgupta et al., 2013; Wang et al., 2013), evaluation of machine translation (Chan and Ng, 2008; Liu et al., 2011), and so on. The SemEval Semantic Textual Similar- ity (STS) task series (Agirre et al., 2012; Agirre et al., 2013; Agirre et al., 2014; Agirre et al., 2015) is a core platform for the task: a publicly available cor- pus of more than 14,000 sentence pairs have been developed over a span of four years with human an- notations of similarity for each pair; and about 300 system runs have been evaluated. In this article, we describe a set of systems that participated in the SemEval-2016 English Seman- tic Textual Similarity (STS) task. Given two En- glish sentences, the objective is to compute their semantic similarity in the range [0, 5], where the score increases with similarity (i.e., 0 indicates no similarity and 5 indicates identical meanings). The official evaluation metric is the Pearson product- moment correlation coefficient with human annota- tions. Our systems leverage different measures of sentence similarity and train ridge regression models that learn to combine predictions from these differ- ent sources using past SemEval data. The best of our three system runs achieves 73.6% with human anno- tations among all submitted systems on five test sets (containing a total of 1186 test pairs). Early work in sentence similarity (Mihalcea et al., 2006; Li et al., 2006; Islam and Inkpen, 2008) estab- lished the basic procedural framework under which most modern algorithms operate: computing sen- tence similarity as a mean of word similarities across the two input sentences. With no human annotated STS dataset available, these algorithms are unsuper- vised and were evaluated extrinsically on tasks like paraphrase detection and textual entailment recog- nition. The SemEval STS task series has made an important contribution through the large annotated dataset, enabling intrinsic evaluation of STS systems and making supervised STS systems a reality. At SemEval 2012–2015, most of the top- performing STS …

Content truncated to 500 words