Demographic Dialectal Variation in Social Media: A Case Study of African-American English

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1119–1130, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Demographic Dialectal Variation in Social Media: A Case Study of African-American English Su Lin Blodgett† Lisa Green∗ Brendan O’Connor† †College of Information and Computer Sciences ∗Department of Linguistics University of Massachusetts Amherst Abstract Though dialectal language is increasingly abundant on social media, few resources exist for developing NLP tools to handle such lan- guage. We conduct a case study of dialectal language in online conversational text by in- vestigating African-American English (AAE) on Twitter. We propose a distantly supervised model to identify AAE-like language from de- mographics associated with geo-located mes- sages, and we verify that this language fol- lows well-known AAE linguistic phenomena. In addition, we analyze the quality of existing language identification and dependency pars- ing tools on AAE-like text, demonstrating that they perform poorly on such text compared to text associated with white speakers. We also provide an ensemble classifier for language identification which eliminates this disparity and release a new corpus of tweets containing AAE-like language. Data and software resources are available at: http://slanglab.cs.umass.edu/TwitterAAE 1 Introduction Owing to variation within a standard language, re- gional and social dialects exist within languages across the world. These varieties or dialects differ from the standard variety in syntax (sentence struc- ture), phonology (sound structure), and the inven- tory of words and phrases (lexicon). Dialect com- munities often align with geographic and sociolog- ical factors, as language variation emerges within distinct social networks, or is affirmed as a marker of social identity. As many of these dialects have traditionally ex- isted primarily in oral contexts, they have histor- ically been underrepresented in written sources. Consequently, NLP tools have been developed from text which aligns with mainstream languages. With the rise of social media, however, dialectal language is playing an increasingly prominent role in online conversational text, for which traditional NLP tools may be insufficient. This impacts many applica- tions: for example, dialect speakers’ opinions may be mischaracterized under social media sentiment analysis or omitted altogether (Hovy and Spruit, 2016). Since this data is now available, we seek to analyze current NLP challenges and extract dialectal language from online data. Specifically, we investigate dialectal language in publicly available Twitter data, focusing on African- American English (AAE), a dialect of Standard American English (SAE) spoken by millions of peo- ple across the United States. AAE is a linguistic variety with defined syntactic-semantic, phonolog- ical, and lexical features, which have been the sub- ject of a rich body of sociolinguistic literature. In addition to the linguistic characterization, reference to its speakers and their geographical location or speech communities is important, especially in light of the historical development of the dialect. Not all African-Americans speak AAE, and not all speakers of AAE are African-American; nevertheless, speak- ers of this variety have close ties with specific com- munities of African-Americans (Green, 2002). Due to its widespread use, established history in the …

Content truncated to 500 words