Streaming First Story Detection with application to Twitter

Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the ACL, pages 181–189, Los Angeles, California, June 2010. c⃝2010 Association for Computational Linguistics Streaming First Story Detection with application to Twitter Saˇsa Petrovi´c School of Informatics University of Edinburgh sasa.petrovic@ed.ac.uk Miles Osborne School of Informatics University of Edinburgh miles@inf.ed.ac.uk Victor Lavrenko School of Informatics University of Edinburgh vlavrenk@inf.ed.ac.uk Abstract With the recent rise in popularity and size of social media, there is a growing need for sys- tems that can extract useful information from this amount of data. We address the prob- lem of detecting new events from a stream of Twitter posts. To make event detection feasi- ble on web-scale corpora, we present an algo- rithm based on locality-sensitive hashing which is able overcome the limitations of traditional approaches, while maintaining competitive re- sults. In particular, a comparison with a state- of-the-art system on the first story detection task shows that we achieve over an order of magnitude speedup in processing time, while retaining comparable performance. Event de- tection experiments on a collection of 160 mil- lion Twitter posts show that celebrity deaths are the fastest spreading news on Twitter. 1 Introduction In the recent years, the microblogging service Twit- ter has become a very popular tool for express- ing opinions, broadcasting news, and simply com- municating with friends. People often comment on events in real time, with several hundred micro-blogs (tweets) posted each second for significant events. Twitter is not only interesting because of this real- time response, but also because it is sometimes ahead of newswire. For example, during the protests fol- lowing Iranian presidential elections in 2009, Iranian people first posted news on Twitter, where they were later picked up by major broadcasting corporations. Another example was the swine flu outbreak when the US Centre for disease control (CDC) used Twit- ter to post latest updates on the pandemic. In ad- dition to this, subjective opinion expressed in posts is also an important feature that sets Twitter apart from traditional newswire. New event detection, also known as first story de- tection (FSD)1 is defined within the topic detection and tracking as one of the subtasks (Allan, 2002). Given a sequence of stories, the goal of FSD is to identify the first story to discuss a particular event. In this context, an event is taken to be something that happens at some specific time and place, e.g., an earthquake striking the town of L’Aquila in Italy on April 6th 2009. Detecting new events from tweets carries additional problems and benefits compared to traditional new event detection from newswire. Problems include a much higher volume of data to deal with and also a higher level of noise. A major benefit of doing new event detection from tweets is the added social component – we can understand the impact an event had and how people reacted to it. The speed and volume at which data is coming from Twitter warrants the use of streaming algo- …

Content truncated to 500 words