Linguistic Redundancy in Twitter

Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 659–669, Edinburgh, Scotland, UK, July 27–31, 2011. c⃝2011 Association for Computational Linguistics Linguistic Redundancy in Twitter Fabio Massimo Zanzotto University of Rome ”Tor Vergata” Rome, Italy zanzotto@info.uniroma2.it Marco Pennacchiotti Yahoo! Labs Sunnyvale, CA, 94089 pennac@yahoo-inc.com Kostas Tsioutsiouliklis Yahoo! Labs Sunnyvale, CA, 94089 kostas@yahoo-inc.com Abstract In the last few years, the interest of the re- search community in micro-blogs and social media services, such as Twitter, is growing ex- ponentially. Yet, so far not much attention has been paid on a key characteristic of micro- blogs: the high level of information redun- dancy. The aim of this paper is to systemat- ically approach this problem by providing an operational definition of redundancy. We cast redundancy in the framework of Textual En- tailment Recognition. We also provide quan- titative evidence on the pervasiveness of re- dundancy in Twitter, and describe a dataset of redundancy-annotated tweets. Finally, we present a general purpose system for identify- ing redundant tweets. An extensive quantita- tive evaluation shows that our system success- fully solves the redundancy detection task, im- proving over baseline systems with statistical significance. 1 Introduction Micro-blogs and social media services, such as Twit- ter, have experienced an exponential growth in the last few years. The interest of the research commu- nity and the industry in these services has followed a similar trend. Web companies such as Google, Ya- hoo, and Bing are integrating more and more social content to their sites. At the same time, the compu- tational linguistic community is getting increasingly interested in studying social and linguistic proper- ties of Twitter and other micro-blogs (Java et al., 2007; Krishnamurthy et al., 2008; Kwak et al., 2010; Zhao et al., 2007; Popescu and Pennacchiotti, 2010; Petrovi´c et al., 2010; Lin et al., 2010; Liu et al., 2010; Ritter et al., 2010). Yet, so far, not much attention has been paid on a key characteristic of micro-blogs: the high level of information redun- dancy. Users often post messages with the same, or very similar, content, especially when reporting or commenting on news and events. For example, the following two tweets are part of a large set of redun- dant tweets issued during the 2010 winter Olympics: (example 1) t1 : “Swiss ski jumper Simon Ammann takes first gold of Vancouver” t2 : “Swiss (Suisse) get the Gold on Normal Hill ski jump. #Vancouver2010” By performing an editorial study (described later in the paper) we discovered that a large part of event- related tweets are indeed redundant. Detecting information redundancy is important for various reasons. First, most applications based on Twitter share the goal of providing tweets that are both informative and diverse, with respect to an initial user information need. For example, Twitter search engines should ideally select the most infor- mative and diverse set of tweets in return to a user query. Similarly, a news web portal that attaches tweets to a given news article should attach those tweets that …

Content truncated to 500 words