Investigating African-American Vernacular English in Transformer-Based Text Generation

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 5877–5883, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 5877 Investigating African-American Vernacular English in Transformer-Based Text Generation Sophie Groenwold∗, Lily Ou∗, Aesha Parekh∗, Samhita Honnavalli∗, Sharon Levy, Diba Mirza, William Yang Wang University of California, Santa Barbara {sophiegroenwold, lilyou, aeshaparekh, shonnavalli}@ucsb.edu {sharonlevy, dimirza, william}@cs.ucsb.edu Abstract The growth of social media has encouraged the written use of African American Vernacular English (AAVE), which has traditionally been used only in oral contexts. However, NLP models have historically been developed using dominant English varieties, such as Standard American English (SAE), due to text corpora availability. We investigate the performance of GPT-2 on AAVE text by creating a dataset of intent-equivalent parallel AAVE/SAE tweet pairs, thereby isolating syntactic structure and AAVE- or SAE-specific language for each pair. We evaluate each sample and its GPT-2 gen- erated text with pretrained sentiment classi- fiers and find that while AAVE text results in more classifications of negative sentiment than SAE, the use of GPT-2 generally increases oc- currences of positive sentiment for both. Ad- ditionally, we conduct human evaluation of AAVE and SAE text generated with GPT-2 to compare contextual rigor and overall quality. 1 Introduction African American Vernacular English (AAVE) is a sociolinguistic variety of American English dis- tinct from Standard American English (SAE) with unique syntactic, semantic, and lexical patterns (Green, 2002; Jones, 2015). Millions of people from predominately Black communities in the United States and Canada use variants of AAVE on a daily basis. Although AAVE has historically been used in spoken contexts, the growing use of social media has encouraged AAVE in written media for which NLP models are increasingly being used. Recent work in Natural Language Generation (NLG) has introduced GPT-2, a Transformer-based language model that generates high-quality, coher- ent text when prompted by arbitrary input (Rad- ford et al., 2019). However, GPT-2 displays bias ∗Equal contribution. towards particular social groups (Solaiman et al., 2019). Sheng et al. (2019) shows that NLG tools are biased with regard to the subject of a sentence when that subject belongs to an underprivileged group, and Shen et al. (2018) tests sentiment analy- sis tools with intent-controlled pairs with varying stylistic inclinations. Studies regarding AAVE have analyzed tasks such as POS tagging (Jørgensen et al., 2016), detecting AAVE syntax (Stewart, 2014), voice recognition and transcription (Dorn, 2019), dependency parsing (Blodgett et al., 2016), and hate speech detection (Sap et al., 2019), but not language generation. Coupled with concerns that NLG tools can be used for generating fake news (Gehrmann et al., 2019) or impersonating internet users (Zellers et al., 2019), it is important that cur- rent work investigates the contexts in which NLG models display bias against certain demographics. In this paper, we examine the bias of GPT-2 text generation against AAVE features. We create a new dataset of AAVE/SAE content-controlled pairs by retrieving AAVE tweets and employing human translators to obtain their SAE counterparts. By doing so, we isolate AAVE syntactic …

Content truncated to 500 words