DLS@CU: Sentence Similarity from Word Alignment and Semantic Vector Composition

Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 148–153, Denver, Colorado, June 4-5, 2015. c⃝2015 Association for Computational Linguistics DLS@CU: Sentence Similarity from Word Alignment and Semantic Vector Composition Md Arafat Sultan†, Steven Bethard‡, Tamara Sumner† †Institute of Cognitive Science and Department of Computer Science University of Colorado Boulder ‡Department of Computer and Information Sciences University of Alabama at Birmingham arafat.sultan@colorado.edu, bethard@cis.uab.edu, sumner@colorado.edu Abstract We describe a set of top-performing systems at the SemEval 2015 English Semantic Textual Similarity (STS) task. Given two English sen- tences, each system outputs the degree of their semantic similarity. Our unsupervised system, which is based on word alignments across the two input sentences, ranked 5th among 73 sub- mitted system runs with a mean correlation of 79.19% with human annotations. We also sub- mitted two runs of a supervised system which uses word alignments and similarities between compositional sentence vectors as its features. Our best supervised run ranked 1st with a mean correlation of 80.15%. 1 Introduction Identification of short text similarity is an important research problem with application in a multitude of areas: natural language processing (machine transla- tion, text summarization), information retrieval (ques- tion answering), education (short answer scoring), and so on. The SemEval Semantic Textual Similarity (STS) task series (Agirre et al., 2012; Agirre et al., 2013; Agirre et al., 2014; Agirre et al., 2015) has become a central platform for the task: a publicly available corpus of more than 14,000 sentence pairs have been developed over the past four years with human annotations of similarity for each pair; and a total of 290 system runs have been evaluated. In this article, we describe a set of systems that were submitted at the SemEval 2015 English STS task (Agirre et al., 2015). Given two English sen- tences, the objective is to compute their semantic similarity in the range [0, 5], where the score in- creases with similarity (i.e., 0 indicates no similarity and 5 indicates identicality). The official evaluation metric was the Pearson correlation coefficient with human annotations. The best of our three system runs achieved the highest mean correlation (80.15%) with human annotations among all submitted systems on five test sets (containing a total of 3000 test pairs). Early work on sentence similarity (Corley and Mi- halcea, 2005; Mihalcea et al., 2006; Li et al., 2006; Islam and Inkpen, 2008) established the basic pro- cedural framework under which most modern algo- rithms operate: computing sentence similarity as a mean of word similarities across the two input sen- tences. With no human annotated STS data set avail- able, these algorithms were unsupervised and were evaluated extrinsically on tasks like paraphrase detec- tion and textual entailment recognition. The SemEval STS task series has made an important contribution through the large annotated data set, enabling intrin- sic evaluation of STS systems and making supervised STS systems a reality. At SemEval 2012, domain-specific training data was provided for most of the test pairs (Agirre et al., 2012) and consequently, supervised …

Content truncated to 500 words