A Paraphrase and Semantic Similarity Detection System for User Generated Short-Text Content on Microblogs

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 2880–2890, Osaka, Japan, December 11-17 2016. A Paraphrase and Semantic Similarity Detection System for User Generated Short-Text Content on Microblogs Kuntal Dey IBM Research India New Delhi, India kuntadey @in.ibm.com Ritvik Shrivastava NSIT Delhi New Delhi, India ritviks.it @nsit.net.in Saroj Kaushik IIT Delhi New Delhi, India saroj @cse.iitd.ac.in Abstract Existing systems deliver high accuracy and F1-scores for detecting paraphrase and semantic sim- ilarity on traditional clean-text corpus. For instance, on the clean-text Microsoft Paraphrase benchmark database, the existing systems attain an accuracy as high as 0.8596. However, ex- isting systems for detecting paraphrases and semantic similarity on user-generated short-text content on microblogs such as Twitter, comprising of noisy and ad hoc short-text, needs signif- icant research attention. In this paper, we propose a machine learning based approach towards this. We propose a set of features that, although well-known in the NLP literature for solving other problems, have not been explored for detecting paraphrase or semantic similarity, on noisy user-generated short-text data such as Twitter. We apply support vector machine (SVM) based learning. We use the benchmark Twitter paraphrase data, released as a part of SemEval 2015, for experiments. Our system delivers a paraphrase detection F1-score of 0.717 and semantic similarity detection F1-score of 0.741, thereby significantly outperforming the existing systems, that deliver F1-scores of 0.696 and 0.724 for the two problems respectively. Our features also allow us to obtain a rank among the top-10, when trained on the Microsoft Paraphrase corpus and tested on the corresponding test data, thereby empirically establishing our approach as ubiquitous across the different paraphrase detection databases. 1 Introduction Detecting paraphrase, and more specifically, given a pair of input natural language texts identifying whether one is a paraphrase of the other, has been a challenging research problem. Over a number of years and volumes of research, the paraphrase detection systems have emerged as robust and well- performing ones, especially for “clean text corpus”, such as the Microsoft Paraphrase Corpus (Dolan et al., 2004). Semantic similarity detection has been another related problem of interest, where the objective is to identify how similar is one text with respect to another (Harispe et al., 2015). Since, paraphrases are expected to have significant semantic similarity (Corley and Mihalcea, 2005) (Mihalcea et al., 2006), systems performing well for paraphrase detection can also be intuitively expected to perform well for semantic similarity detection of a pair of text inputs. It is worth noting that, for benchmark “clean text corpus” data such as the Microsoft Paraphrase Corpus database, the paraphrase detection systems deliver an impressive performance, with a high F-score of 0.8596. While the literature around paraphrase and semantic similarity detection has matured along the direc- tions of clean text corpus, much work remains to be done to attain commendable performances in the paradigm of noisy short-text inputs, such as user-generated short-text content found on Twitter. Some initial work has been recently carried out on the benchmark SemEval 2015 …

Content truncated to 500 words