DLS@CU-CORE: A Simple Machine Learning Model of Semantic Textual Similarity

Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 1: Proceedings of the Main Conference and the Shared Task, pages 176–180, Atlanta, Georgia, June 13-14, 2013. c⃝2013 Association for Computational Linguistics DLS@CU-CORE: A Simple Machine Learning Model of Semantic Textual Similarity Md. Arafat Sultan, Steven Bethard, Tamara Sumner Institute of Cognitive Science and Department of Computer Science University of Colorado, Boulder, CO 80309 {arafat.sultan, steven.bethard, sumner}@colorado.edu Abstract We present a system submitted in the Semantic Textual Similarity (STS) task at the Second Joint Conference on Lexical and Computa- tional Semantics (*SEM 2013). Given two short text fragments, the goal of the system is to determine their semantic similarity. Our sys- tem makes use of three different measures of text similarity: word n-gram overlap, character n-gram overlap and semantic overlap. Using these measures as features, it trains a support vector regression model on SemEval STS 2012 data. This model is then applied on the STS 2013 data to compute textual similarities. Two different selections of training data result in very different performance levels: while a cor- relation of 0.4135 with gold standards was ob- served in the official evaluation (ranked 63rd among all systems) for one selection, the other resulted in a correlation of 0.5352 (that would rank 21st). 1 Introduction Automatically identifying the semantic similarity between two short text fragments (e.g. sentences) is an important research problem having many im- portant applications in natural language processing, information retrieval, and digital education. Exam- ples include automatic text summarization, question answering, essay grading, among others. However, despite having important applications, semantic similarity identification at the level of short text fragments is a relatively recent area of in- vestigation. The problem was formally brought to attention and the first solutions were proposed in 2006 with the works reported in (Mihalcea et al., 2006) and (Li et al., 2006). Work prior to these fo- cused primarily on large documents (or individual words) (Mihalcea et al., 2006). But the sentence- level granularity of the problem is characterized by factors like high specificity and low topicality of the expressed information, and potentially small lexical overlap even between very similar texts, asking for an approach different from those that were designed for larger texts. Since its inception, the problem has seen a large number of solutions in a relatively small amount of time. The central idea behind most solutions is the identification and alignment of semantically similar or related words across the two sentences, and the aggregation of these similarities to generate an over- all similarity score (Mihalcea et al., 2006; Islam and Inkpen, 2008; Šarić et al., 2012). The Semantic Textual Similarity task (STS) or- ganized as part of the Semantic Evaluation Exer- cises (see (Agirre et al., 2012) for a description of STS 2012) provides a common platform for evalua- tion of such systems via comparison with human- annotated similarity scores over a large dataset. In this paper, we present a system which was submitted in STS 2013. Our system is based on very …

Content truncated to 500 words