SemEval-2015 Task 1: Paraphrase and Semantic Similarity in Twitter (PIT)

Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 1–11, Denver, Colorado, June 4-5, 2015. c⃝2015 Association for Computational Linguistics SemEval-2015 Task 1: Paraphrase and Semantic Similarity in Twitter (PIT) Wei Xu and Chris Callison-Burch University of Pennsylvania Philadelphia, PA, USA xwe, ccb@cis.upenn.edu William B. Dolan Microsoft Research Redmond, WA, USA billdol@microsoft.com Abstract In this shared task, we present evaluations on two related tasks Paraphrase Identification (PI) and Semantic Textual Similarity (SS) sys- tems for the Twitter data. Given a pair of sentences, participants are asked to produce a binary yes/no judgement or a graded score to measure their semantic equivalence. The task features a newly constructed Twitter Para- phrase Corpus that contains 18,762 sentence pairs. A total of 19 teams participated, sub- mitting 36 runs to the PI task and 26 runs to the SS task. The evaluation shows encourag- ing results and open challenges for future re- search. The best systems scored a F1-measure of 0.674 for the PI task and a Pearson corre- lation of 0.619 for the SS task respectively, comparing to a strong baseline using logis- tic regression model of 0.589 F1 and 0.511 Pearson; while the best SS systems can of- ten reach >0.80 Pearson on well-formed text. This shared task also provides insights into the relation between the PI and SS tasks and sug- gests the importance to bringing these two re- search areas together. We make all the data, baseline systems and evaluation scripts pub- licly available.1 1 Introduction The ability to identify paraphrases, i.e. alternative expressions of the same (or similar) meaning, and the degree of their semantic similarity has proven useful for a wide variety of natural language pro- cessing applications (Madnani and Dorr, 2010). It 1http://www.cis.upenn.edu/˜xwe/ semeval2015pit/ is particularly useful to overcome the challenge of high redundancy in Twitter and the sparsity inherent in their short texts (e.g. oscar nom’d doc ↔ Oscar- nominated documentary; some1 shot a cop ↔ some- one shot a police). Emerging research shows para- phrasing techniques applied to Twitter data can im- prove tasks like first story detection (Petrovi´c et al., 2012), information retrieval (Zanzotto et al., 2011) and text normalization (Xu et al., 2013; Wang et al., 2013). Previously, many researchers have investigated ways of automatically detecting paraphrases on more formal texts, like newswire text. The ACL Wiki2 gives an excellent summary of the state-of- the-art paraphrase identification techniques. These can be categorized into supervised methods (Qiu et al., 2006; Wan et al., 2006; Das and Smith, 2009; Socher et al., 2011; Blacoe and Lapata, 2012; Mad- nani et al., 2012; Ji and Eisenstein, 2013) and unsu- pervised methods (Mihalcea et al., 2006; Rus et al., 2008; Fernando and Stevenson, 2008; Islam and Inkpen, 2007; Hassan and Mihalcea, 2011). A few recent studies have highlighted the potential and importance of developing paraphrase identification (Zanzotto et al., 2011; Xu et al., 2013) and semantic similarity techniques (Guo and Diab, 2012) specif- ically for tweets. They also indicated that the very informal …

Content truncated to 500 words