TKLBLIIR: Detecting Twitter Paraphrases with TweetingJay

Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 70–74, Denver, Colorado, June 4-5, 2015. c⃝2015 Association for Computational Linguistics TKLBLIIR: Detecting Twitter Paraphrases with TweetingJay Mladen Karan1, Goran Glavaˇs1, Jan ˇSnajder1, Bojana Dalbelo Baˇsi´c1, Ivan Vuli´c2, and Marie-Francine Moens2 1University of Zagreb, Faculty of Electrical Engineering and Computing Text Analysis and Knowledge Engineering Lab, Unska 3, 10000 Zagreb, Croatia {goran.glavas, mladen.karan, jan.snajder, bojana.dalbelo}@fer.hr 2KU Leuven, Department of Computer Science Language Intelligence & Information Retrieval Group, Celestijnenlaan 200A, Leuven, Belgium {ivan.vulic, sien.moens}@cs.kuleuven.be Abstract When tweeting on a topic, Twitter users often post messages that convey the same or similar meaning. We describe TweetingJay, a system for detecting paraphrases and semantic simi- larity of tweets, with which we participated in Task 1 of SemEval 2015. TweetingJay uses a supervised model that combines semantic over- lap and word alignment features, previously shown to be effective for detecting semantic textual similarity. TweetingJay reaches 65.9% F1-score and ranked fourth among the 18 par- ticipating systems. We additionally provide an analysis of the dataset and point to some peculiarities of the evaluation setup. 1 Introduction Recognizing tweets that convey the same meaning (paraphrases) or similar meaning is useful in applica- tions such as event detection (Petrovi´c et al., 2012), tweet summarization (Yang et al., 2011), and tweet retrieval (Naveed et al., 2011). Paraphrase detection in tweets is a more challenging task than paraphrase detection in other domains such as news (Xu et al., 2013). Besides brevity (max. 140 characters), tweets exhibit all the irregularities typical of social media text (Baldwin et al., 2013), such as informality, un- grammaticality, disfluency, and excessive use of jar- gon. In this paper we present the TweetingJay system for detecting paraphrases in tweets, with which we participated in Task 1 of SemEval 2015 evaluation exercise (Xu et al., 2015). Our system builds on findings from a large body of work on semantic tex- tual similarity (STS) (ˇSari´c et al., 2012; Sultan et al., 2014) and recent breakthroughs in distributed word representations (Mikolov et al., 2013a). We design a set of measures that capture the semantic similarity of tweets and train a support vector machine (SVM) using these measures as features. Positioning of our system at rank four among 18 teams, with only point and a half lower performance compared to the the best-performing system, suggests that STS measures are useful for detecting paraphrases in Twitter. We make our system freely available.1 Besides providing the description of the Tweeting- Jay system, in this paper we analyze the evaluation setup, with special focus on the provided dataset and its subsets (train, validation, and test), and discuss the stability of the evaluation results. 2 Related Work There is a large body of work on automated para- phrase detection; see (Madnani and Dorr, 2010) for a comprehensive overview. The majority of research efforts focus on detecting paraphrases in standard texts such as news (Das and Smith, 2009; Madnani et al., 2012) or artificially generated text (Madnani et al., 2012). State-of-the-art approaches …

Content truncated to 500 words