Samsung: Align-and-Differentiate Approach to Semantic Textual Similarity

Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 172–177, Denver, Colorado, June 4-5, 2015. c⃝2015 Association for Computational Linguistics Samsung: Align-and-Differentiate Approach to Semantic Textual Similarity Lushan Han, Justin Martineau, Doreen Cheng and Christopher Thomas Samsung Research America 665 Clyde Avenue Mountain View, CA 94043, USA {lushan.han, justin.m, doreen.c, c2.thomas}@samsung.com Abstract This paper describes our Align-and- Differentiate approach to the SemEval 2015 Task 2 competition for English Seman- tic Textual Similarity (STS) systems. Our submission achieved the top place on two of the five evaluation datasets. Our team placed 3rd among 28 participating teams, and our three runs ranked 4th, 6th and 7th among the 73 runs submitted by the 28 teams. Our approach improves upon the UMBC PairingWords system by semantically differentiating distributionally similar terms. This novel addition improves results by 2.5 points on the Pearson correlation measure. 1 Introduction Since its inception in 2012, the annual Semantic Textual Similarity (STS) task has attracted and in- creasing amount of interest in the NLP community. The task is to measure the semantic similarity be- tween two sentences using a scale ranging from 0 to 5 (Agirre et al., 2012; Agirre et al., 2013; Agir- rea et al., 2014). In this task, 0 means unrelated and 5 means complete semantic equivalence. For example, the sentence “China’s new PM rejects US hacking claims” is semantically equivalent to the sentence “China Premier Li rejects ‘groundless’ US hacking accusations” even though there are many word level differences between the two sentences. Improvements in the STS task can advance or benefit many research areas, such as paraphrase recognition (Dolan et al., 2004), automatic ma- chine translation evaluation (Kauchak and Barzi- lay, 2006), ontology mapping and schema matching (Han, 2014), Twitter search (Sriram et al., 2010), im- age retrieval by captions (Coelho et al., 2004) and information retrieval in general. Measuring semantic similarity is difficult because it is relatively easy to express the same idea in very different ways. Both word choice and word order can have a great impact on the semantics of a sen- tence, or not at all. For example, the sentences “A woman is playing piano on the street” and “A lady is playing violin on the street” have a semantic sim- ilarity score of only 2, because pianos are not vio- lins so the two events in the sentences must be dif- ferent. This is problematic because common solu- tions, such as bag-of-words representations, parse trees, and word alignments measure word choice and word order. We improve upon existing word choice approaches with better measures to semantically dif- ferentiate distributionally similar terms, and by us- ing these measures to also improve the word align- ment. Our solution is an Align-and-Differentiate ap- proach, in which we greedily align words between sentences, before penalizing non-matching words in the differentiate-phase. Our system improves upon the successful UMBC PairingWords system by about 2 points of Pearson’s Correlation measure. The success of the PairingWords system is largely due to their high-quality distributional …

Content truncated to 500 words