Does Robustness Improve Fairness? Approaching Fairness with Word Substitution Robustness Methods for Text Classification

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3320–3331 August 1–6, 2021. ©2021 Association for Computational Linguistics 3320 Does Robustness Improve Fairness? Approaching Fairness with Word Substitution Robustness Methods for Text Classification Yada Pruksachatkun ∗ Amazon Alexa ypruksac@amazon.com Rahul Gupta Amazon Alexa gupra@amazon.com Satyapriya Krishna ∗ Amazon Alexa satyapk@amazon.com Jwala Dhamala Amazon Alexa jddhamal@amazon.com Kai Wei Chang UCLA, Amazon Alexa kwchang@cs.ucla.edu Abstract Existing bias mitigation methods to reduce disparities in model outcomes across cohorts have focused on data augmentation, debiasing model embeddings, or adding fairness-based optimization objectives during training. Sep- arately, certified word substitution robustness methods have been developed to decrease the impact of spurious features and synonym sub- stitutions on model predictions. While their end goals are different, they both aim to en- courage models to make the same prediction for certain changes in the input. In this paper, we investigate the utility of certified word sub- stitution robustness methods to improve equal- ity of odds and equality of opportunity on mul- tiple text classification tasks. We observe that certified robustness methods improve fairness, and using both robustness and bias mitigation methods in training results in an improvement in both fronts. 1 Introduction As natural language processing (NLP) technolo- gies are increasingly used in essential real-world applications, such as social media, healthcare, per- sonal assistants and law (He et al., 2020; Ahmad et al., 2020), it is important to ensure these systems do not create unintended outcomes for end-users or offer disparate experiences to customers from diverse backgrounds. This includes ensuring that model performance does not significantly differ across people belonging to different cohorts, such as different gender or race groups. A major subset of industry NLP applications lies in text classification, such as domain and in- tent classification in voice assistants (Su et al., 2018) or code tagging in healthcare (Kemp et al., 2019). In this study, we focus on toxicity classifi- ∗* Equal contribution cation (Dixon et al., 2018) and occupation classifi- cation of Wikipedia biographies (De-Arteaga et al., 2019). For toxicity classification, ensuring fairness means ensuring that a model can identify toxicity to a similar accuracy across all examples regard- less of the protected groups present in the example. Past studies (e.g, (Dixon et al., 2018; Zhang et al., 2020; Zhao and Chang, 2020)) have shown that toxicity classification models will falsely classify text containing certain protected attributes as toxic. Leading social media platforms and internet com- panies use toxicity classification models for con- tent moderation (Gorwa et al., 2020), thus having bias in such models can lead to increased silencing of under-served groups. Similarly, for occupation classification, a fair model should correctly iden- tify occupations given a biography, regardless of the protected group that a person belongs to (De- Arteaga et al., 2019). Recently, several studies have demonstrated so- cietal bias in NLP systems (Hutchinson et al., 2020; Tan and Celis, 2019; Liang et al., 2020) and vari- ous approaches have been proposed to mitigate the bias. These approaches include creating balanced datasets …

Content truncated to 500 words