Confounds and Consequences in Geotagged Twitter Data

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2138–2148, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. Confounds and Consequences in Geotagged Twitter Data Umashanthi Pavalanathan and Jacob Eisenstein School of Interactive Computing Georgia Institute of Technology Atlanta, GA 30308 {umashanthi + jacobe}@gatech.edu Abstract Twitter is often used in quantitative stud- ies that identify geographically-preferred topics, writing styles, and entities. These studies rely on either GPS coordinates at- tached to individual messages, or on the user-supplied location field in each profile. In this paper, we compare these data ac- quisition techniques and quantify the bi- ases that they introduce; we also measure their effects on linguistic analysis and text- based geolocation. GPS-tagging and self- reported locations yield measurably dif- ferent corpora, and these linguistic differ- ences are partially attributable to differ- ences in dataset composition by age and gender. Using a latent variable model to induce age and gender, we show how these demographic variables interact with geography to affect language use. We also show that the accuracy of text-based geolocation varies with population demo- graphics, giving the best results for men above the age of 40. 1 Introduction Social media data such as Twitter is frequently used to identify the unique characteristics of geographical regions, including topics of inter- est (Hong et al., 2012), linguistic styles and di- alects (Eisenstein et al., 2010; Gonc¸alves and S´anchez, 2014), political opinions (Caldarelli et al., 2014), and public health (Broniatowski et al., 2013). Social media permits the aggregation of datasets that are orders of magnitude larger than could be assembled via traditional survey tech- niques, enabling analysis that is simultaneously fine-grained and global in scale. Yet social media is not a representative sample of any “real world” population, aside from social media itself. Using social media as a sample therefore risks introduc- ing both geographic and demographic biases (Mis- love et al., 2011; Hecht and Stephens, 2014; Lon- gley et al., 2015; Malik et al., 2015). This paper examines the effects of these bi- ases on the geo-linguistic inferences that can be drawn from Twitter. We focus on the ten largest metropolitan areas in the United States, and con- sider three sampling techniques: drawing an equal number of GPS-tagged tweets from each area; drawing a county-balanced sample of GPS-tagged messages to correct Twitter’s urban skew (Hecht and Stephens, 2014); and drawing a sample of location-annotated messages, using the location field in the user profile. Leveraging self-reported first names and census statistics, we show that the age and gender composition of these datasets dif- fer significantly. Next, we apply standard methods from the lit- erature to identify geo-linguistic differences, and test how the outcomes of these methods depend on the sampling technique and on the underlying demographics. We also test the accuracy of text- based geolocation (Cheng et al., 2010; Eisenstein et al., 2010) in each dataset, to determine whether the accuracies reported in recent work will gener- alize to more balanced samples. The paper reports …

Content truncated to 500 words