A Comparative Study on the Impact of Model Compression Techniques on Fairness in Language Models

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 15762–15782 July 9-14, 2023 ©2023 Association for Computational Linguistics A Comparative Study on the Impact of Model Compression Techniques on Fairness in Language Models Krithika Ramesh Microsoft Research kramesh.tlw@gmail.com Arnav Chavan∗ Indian Institute of Technology, Dhanbad arnavchavan04@gmail.com Shrey Pandit∗ BITS, Pilani pandit.shrey.01@gmail.com Sunayana Sitaram Microsoft Research sunayana.sitaram@microsoft.com Abstract Compression techniques for deep learning have become increasingly popular, particularly in settings where latency and memory constraints are imposed. Several methods, such as prun- ing, distillation, and quantization, have been adopted for compressing models, each provid- ing distinct advantages. However, existing lit- erature demonstrates that compressing deep learning models could affect their fairness. Our analysis involves a comprehensive evaluation of pruned, distilled, and quantized language models, which we benchmark across a range of intrinsic and extrinsic metrics for measur- ing bias in text classification. We also investi- gate the impact of using multilingual models and evaluation measures. Our findings high- light the significance of considering both the pre-trained model and the chosen compression strategy in developing equitable language tech- nologies. The results also indicate that com- pression strategies can have an adverse effect on fairness measures. 1 Introduction Despite their increasing popularity, machine learn- ing models have been known to exhibit biases in their outputs, present privacy risks, and have poten- tially negative environmental consequences from their training and deployment. (Bender et al., 2021; Talat et al., 2022). Language models suffer from biases that result in unequal resource distributions (allocational harms), in addition to the undesired tendency to reproduce biases and stereotypes in content that is reflective of hegemonic worldviews (representational harms). Although measures have been proposed in tasks such as text classi- fication (Czarnowska et al., 2021) to investigate the disparate allocational treatment of different classes, much of the research on fairness in language mod- els centers on addressing representational harms *Equal contribution (Blodgett et al., 2020). The potential of these mod- els to further stigmatize marginalized communi- ties is demonstrated in (Dressel and Farid, 2018), which illustrates how recidivism prediction systems are biased against black defendants, who have a higher baseline risk for repeat offences. Biases are also prevalent in computer vision applications such as facial recognition technologies. Within NLP, (Bolukbasi et al., 2016), one of the first forays that studied this phenomenon in language, noted that word embeddings contained stereotypical as- sociations with respect to gender. Language mod- els can exhibit biases toward different dialects for tasks like toxicity and hate speech detection (Garg et al., 2022; Sap et al., 2019), generate stereotypi- cal representations and narratives (Lucy and Bam- man, 2021), and are capable of the outright erasure of underrepresented identities (Dev et al., 2021). Compressed models that are biased may have detri- mental consequences in the real world, as they are typically deployed on edge devices, which can fur- ther disadvantage communities without access to other forms of technology. Consequently, these issues have compelled a shift towards developing more inclusive …

Content truncated to 500 words