Evaluating and Mitigating Inherent Linguistic Bias of African American English through Inference

Proceedings of the 29th International Conference on Computational Linguistics, pages 1442–1454 October 12–17, 2022. 1442 Evaluating and Mitigating Inherent Linguistic Bias of African American English through Inference Jamell Dacon∗ Haochen Liu Michigan State UniversityJiliang Tang {daconjam, liuhoac1, tangjili}@msu.edu Abstract Recent studies show that NLP models trained on standard English texts tend to produce bi- ased outcomes against underrepresented En- glish varieties. In this work, we conduct a pi- oneering study of the English variety use of African American English (AAE) in NLI task. First, we propose CODESWITCH, a greedy uni- directional morphosyntactically-informed rule- based translation method for data augmenta- tion. Next, we use CODESWITCH to present a preliminary study to determine if demographic language features do in fact influence models to produce false predictions. Then, we con- duct experiments on two popular datasets and propose two simple, yet effective and general- izable debiasing methods. Our findings show that NLI models (e.g. BERT) trained under our proposed frameworks outperform traditional large language models while maintaining or even improving the prediction performance. In addition, we intend to release CODESWITCH, in hopes of promoting dialectal language diver- sity in training data to both reduce the discrim- inatory societal impacts and improve model robustness of downstream NLP tasks. 1 Introduction In recent years, social media has become a piv- otal tool its users to express their thoughts, feel- ings, and opinions on similar interests (Dacon and Tang, 2021). Typically, Standard American English (SAE), a high-resource language (HRL) is often used in formal communication, whereas African American English (AAE)1 is primarily spoken in ∗Corresponding author: Jamell Dacon 1This English language variety has had several names within the last decades such as African American Vernacular English (AAVE), African American Language (AAL), Black English, Ebonics, Non-standard English, Northern Negro En- glish and Black English Vernacular (BEV) (Bailey et al., 1998; Green, 2002; Bland-Stewart, 2005; King, 2020). However, it is now commonly referred to as African American English (AAE), an English language variety. the United States and is often heavily and explic- itly used on social media platforms such as Twitter (Field et al., 2021; Blodgett et al., 2020). In particular, AAE is an English language vari- ety and can be considered to be a low-resource lan- guage (LRL) that is neither spoken by all African Americans or individuals who identify as BIPOC (Black, Indigenous, or People of Color), nor is it spoken only by African Americans or BIPOC individuals (Field et al., 2021; Dacon, 2022; Bland- Stewart, 2005). However, most dominant AAE speakers reside in diglossic communities and are able to code-switch, speaking both SAE and AAE. In linguistics, code-switching also referred to as language alternation is the ability of a speaker to al- ternate between two or more languages or language varieties within a particular conversation (Young and Barrett, 2018; Gardner-Chloros et al., 2009; DeBose, 1992; Young, 2009; Dacon, 2022). Thus, we refer to code-switching as switching among di- alects, and/or language styles. For example, bi- dialectal AAE speakers are often able to code- switch between …

Content truncated to 500 words