Syndromic Surveillance using Generic Medical Entities on Twitter

Syndromic Surveillance using Generic Medical Entities on Twitter Pin Huang♦, Andrew MacKinlay♦♠ and Antonio Jimeno Yepes♦ ♦ IBM Research – Australia, Melbourne, VIC, Australia ♠ Dept of Computing and Information Systems, University of Melbourne, Australia pinhuang@outlook.com, {admackin,antonio.jimeno}@au1.ibm.com Abstract Public health surveillance is challenging due to difficulties accessing medical data in real-time. We present a novel, ef- fective and computationally inexpensive method for syndromic surveillance using Twitter data. The proposed method uses a regression model on a database previ- ously built using named entity recognition to identify mentions of symptoms, disor- ders and pharmacological substances over GNIP Decahose Twitter data. The result of our method is compared to the reported weekly flu and Lyme disease rates from the US Center of Disease Control and Pre- vention (CDC) website. Our method pre- dicts the 2014 CDC reported flu preva- lence with 94.9% Spearman correlation using 2012 and 2013 CDC flu statistics as training data, and the CDC Lyme dis- ease rate for July to December 2014 with 89.6% Spearman correlation. It also pre- dicts the prevalences for the same diseases and time periods using the Twitter data from the previous week with 93.31% and 86.9% Spearman correlations respectively. 1 Introduction Real-time public health surveillance for tasks such as syndromic surveillance is challenging due to difficulties accessing medical data. Twitter is a social media platform in which people share their views, opinions and their lives. Data from Twit- ter is accessible in real-time and it could poten- tially be used for syndromic surveillance. Even if only a small portion of the tweets contains po- tentially information about the health of Twitter users (Jimeno-Yepes et al., 2015a), there is still a large volume of data that could be useful for public health surveillance. Several approaches to predict flu prevalences from Twitter data already exist. These approaches either rely on topic modelling (e.g. Latent Dirich- let Allocation (LDA) (Blei et al., 2003)) (Paul and Dredze, 2012; Paul and Dredze, 2011) or rely on regression models on keyword frequency (Culotta, 2010a; Culotta, 2010b). The topic modelling approach for flu preva- lence prediction requires manually labelling a large number of tweets (e.g. 5,128 tweets) that are used to train a Support Vector Machine (SVM) (Joachims, 1999) classifier applied on 11.7 mil- lion messages. The predictions on the tweets are applied on a LDA based topic model to over millions of tweets (Paul and Dredze, 2012; Paul and Dredze, 2011). Regression approaches (Cu- lotta, 2010a; Culotta, 2010b) require prior knowl- edge to develop a keyword list {flu,cough,sore throat,headache} that could identify tweets rele- vant to flu. In this paper, we propose an effective and computationally efficient alternative for disease prevalence prediction based on an already exist- ing database developed by (Jimeno-Yepes et al., 2015b). Our approach to predict disease preva- lence does not require manual labelling of Twit- ter posts to determine whether the posts are re- lated to a particular disease or not. Our train- ing dataset uses aggregated weekly term frequen- cies, so it is less …

Content truncated to 500 words