Your fairness may vary: Pretrained language model fairness in toxic text classification

Findings of the Association for Computational Linguistics: ACL 2022, pages 2245 - 2262 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Your fairness may vary: Pretrained language model fairness in toxic text classification Ioana Baldini Dennis Wei Mikhail YurochkinKarthikeyan Natesan Ramamurthy Moninder Singh IBM Research {ioana,dwei,knatesa,moninder}@us.ibm.com mikhail.yurochkin@ibm.com Abstract The popularity of pretrained language models in natural language processing systems calls for a careful evaluation of such models in down-stream tasks, which have a higher potential for societal impact. The evaluation of such systems usually focuses on accuracy measures. Our findings in this paper call for attention to be paid to fairness measures as well. Through the analysis of more than a dozen pretrained language models of varying sizes on two toxic text classification tasks (English), we demonstrate that focusing on accuracy measures alone can lead to models with wide variation in fairness characteristics. Specifically, we observe that fairness can vary even more than accuracy with increasing training data size and different random initializations. At the same time, we find that little of the fairness variation is explained by model size, despite claims in the literature. To improve model fairness without retraining, we show that two post-processing methods developed for structured, tabular data can be successfully applied to a range of pretrained language models. Warning: This paper contains samples of offensive text. 1 Introduction Pre-trained, bidirectional language models (De- vlin et al., 2019; Liu et al., 2019; Radford et al., 2019; Clark et al., 2020; He et al., 2021)1 have revolutionized natural language processing (NLP) research. LMs have provided a route to signifi- cant performance increases in several NLP tasks as demonstrated by NLP leaderboards (Rajpurkar et al., 2018; Wang et al., 2019a,b; AI2, 2021). More importantly, LMs have been applied to prac- tical problems, leading to improved results for web search (Nayak, 2019) and have become an asset in 1We use the acronym LM(s) to refer to language model(s) throughout the paper. fields such as medical evidence inference (Lehman et al., 2019; Subramanian et al., 2020) and chem- istry (Schwaller et al., 2021). While the progress in NLP tasks due to LMs is clear, the reasons behind this success are not as well understood (Rogers et al., 2021; McCoy et al., 2019), and there are also important downsides. In particular, several stud- ies have documented the bias of LMs (Bolukbasi et al., 2016; Hutchinson et al., 2020; Webster et al., 2020; Borkan et al., 2019; de Vassimon Manela et al., 2021) and others discuss potential societal harms (Blodgett et al., 2020; Bender et al., 2021) for individuals or groups. We use the term bias to refer to systematic disparity in representation or outcomes for individuals based on their mem- bership in certain protected groups such as gender, race, and ethnicity. In this work, we focus on one important ap- plication of fine-tuned LMs, toxic text classifi- cation. Text toxicity predictors are already used in deployed systems (Perspective API, 2021) and they are a crucial component for content modera- tion since …

Content truncated to 500 words