Effects of Sampling on Twitter Trend Detection

Effects of Sampling on Twitter Trend Detection Andrew Yates1, Alek Kolcz2, Nazli Goharian1, Ophir Frieder1 1Information Retrieval Lab, Department of Computer Science, Georgetown University 2Twitter, Inc. {andrew,nazli,ophir}@ir.cs.georgetown.edu Abstract Much research has focused on detecting trends on Twitter, including health-related trends such as mentions of Influenza-like illnesses or their symptoms. The majority of this research has been conducted using Twitter’s public feed, which includes only about 1% of all public tweets. It is unclear if, when, and how using Twitter’s 1% feed has affected the evaluation of trend detection methods. In this work we use a larger feed to investigate the effects of sampling on Twitter trend detection. We focus on using health-related trends to estimate the prevalence of Influenza-like illnesses based on tweets, and use ground truth obtained from the CDC and Google Flu Trends to explore how the prevalence estimates degrade when moving from a 100% to a 1% sample. We find that using the public 1% sample is unlikely to substantially harm ILI estimates made at the national level, but can cause poor performance when estimates are made at the city level. Keywords: LR Infrastructures and Architectures, Social Media, Trend Detection 1. Introduction Much work has focused on detecting health-related Twit- ter trends and, in particular, trends related to the prevalence of Influenza-like illnesses (ILI). Such trends are useful be- cause they can provide prevalence estimates more quickly than formal reporting systems. The majority of this trend detection research has been conducted on a 1% sample of Twitter’s public feed, however, and it is unclear whether this has biased evaluation results. In this work we use a complete feed of all public tweets to investigate the effects of using the 1% sample. We evaluate the performance of a trend detection method using progressively larger sample sizes to characterize how large the sample must be before evaluation results begin to converge. Influenza-like illnesses are unusual in that they follow a seasonal pattern, affect a relatively large percentage of the population, and are closely monitored by organizations such as the ECDC (European Centre for Disease Preven- tion and Control) and CDC (the United States’ Centers for Disease Control and Prevention). Publicly available preva- lence data has been widely used to evaluate how well trend detection methods can estimate ILI prevalence, making ILI prevalence estimation well-suited for evaluating the perfor- mance of trend detection methods. We use CDC and GFT (Google Flu Trends) (Ginsberg et al., 2009) ILI prevalence data to evaluate how methods for detecting health-related Twitter trends (Yates et al., 2014) perform when used with varying sample sizes. Our contri- butions are: • an exploration of how sampling affects flu detection, and health-trend detection in general, on Twitter; • an analysis of the degree to which inaccuracies intro- duced by sampling can be explained by the number of tweets being analyzed; and • suggestions for reducing the effects of sampling when using a sampled Twitter feed. 2. Related Work Several researchers have investigated how Twitter chooses the tweets in …

Content truncated to 500 words