Mapping Dialectal Variation by Querying Social Media

Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 98–106, Gothenburg, Sweden, April 26-30 2014. c⃝2014 Association for Computational Linguistics Mapping dialectal variation by querying social media Gabriel Doyle Department of Linguistics University of California, San Diego La Jolla, CA, USA 92093-0108 gdoyle@ucsd.edu Abstract We propose a Bayesian method of esti- mating a conditional distribution of data given metadata (e.g., the usage of a di- alectal variant given a location) based on queries from a big data/social me- dia source, such as Twitter. This distri- bution is structurally equivalent to those built from traditional experimental meth- ods, despite lacking negative examples. Tests using Twitter to investigate the ge- ographic distribution of dialectal forms show that this method can provide distri- butions that are tightly correlated with ex- isting gold-standard studies at a fraction of the time, cost, and effort. 1 Introduction Social media provides a linguist with a new data source of unprecedented scale, opening novel av- enues for research in empirically-driven areas, such as corpus and sociolinguistics. Extracting the right information from social media, though, is not as straightforward as in traditional data sources, as the size and format of big data makes it too un- wieldy to observe as a whole. Researchers often must interact with big data through queries, which produce only positive results, those matching the search term. At best, this can be augmented with a set of “absences” covering results that do not match the search term, but explicit negative data (e.g., confirmation that a datapoint could never match the search term) does not exist. In addition to the lack of explicit negative data, query-derived data has a conditional distribution that reverses the dependent and independent variables compared to traditional data sources, such as sociolinguistic in- terviews. This paper proposes a Bayesian method for overcoming these two difficulties, allowing query- derived data to be applied to traditional problems without requiring explicit negative data or the abil- ity to view the entire dataset at once. The test case in this paper is dialect geography, where the pos- itive data is the presence of a dialectal word or phrase in a tweet, and the metadata is the location of the person tweeting it. However, the method is general and applies to any queryable big data source that includes metadata about the user or set- ting that generated the data. The key to this method lies in using an indepen- dent query to estimate the overall distribution of the metadata. This estimated distribution corrects for non-uniformity in the data source, enabling the reversal of the conditionality on the query-derived distribution to convert it to the distribution of in- terest. Section 2 explains the mathematical core of the Bayesian analysis. Section 3 implements this anal- ysis for Twitter and introduces an open-source program for determining the geographic distri- bution of tweets. Section 4 tests the method on problems in linguistic geography and shows that its results are well-correlated with those of tradi- …

Content truncated to 500 words