Stereotype and Skew: Quantifying Gender Bias in Pre-trained and Fine-tuned Language Models

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 2232–2242 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 2232 Stereotype and Skew: Quantifying Gender Bias in Pre-trained and Fine-tuned Language Models Daniel de Vassimon Manela† David Errington† Thomas Fisher† Boris van Breugel† Pasquale Minervini University College London {daniel.manela.19,david.errington.19,thomas.fisher.19, boris.breugel.19,p.minervini}@ucl.ac.uk Abstract This paper proposes two intuitive metrics, skew and stereotype, that quantify and analyse the gender bias present in contextual language models when tackling the WinoBias pronoun resolution task. We find evidence that gen- der stereotype correlates approximately nega- tively with gender skew in out-of-the-box mod- els, suggesting that there is a trade-off between these two forms of bias. We investigate two methods to mitigate bias. The first approach is an online method which is effective at remov- ing skew at the expense of stereotype. The sec- ond, inspired by previous work on ELMo, in- volves the fine-tuning of BERT using an aug- mented gender-balanced dataset. We show that this reduces both skew and stereotype rel- ative to its unaugmented fine-tuned counter- part. However, we find that existing gender bias benchmarks do not fully probe profes- sional bias as pronoun resolution may be ob- fuscated by cross-correlations from other man- ifestations of gender prejudice. Our code is available online. 1 Introduction Transformer-Based Transfer Learning models for NLP – referred to henceforth as TBTL models for brevity – such as BERT (Devlin et al., 2018), RoBERTa (Liu et al., 2019), and ALBERT (Lan et al., 2020) perform well on a variety of NLP tasks with minimal fine-tuning. However, prior to fine- tuning, TBTL models require a vast amount of data to train (Shoeybi et al., 2019). This training is only performed once, with users downloading and fine-tuning such language models to their specific task. In doing so, we are trusting large tech compa- nies to train the base model responsibly since we have no control over this. This seems inherently undemocratic. We ideally want these models to be †Equal contribution. free from unwanted bias and whilst it is true that they exhibit less gender bias than static word em- beddings (Sun et al., 2019), they are by no means immune to this problem (Lu et al., 2018). As TBTL models become increasingly preva- lent in our everyday lives, we want to avoid such prejudices influencing decision making. Examples where this is important include automatic resume filtering (Dastin, 2018) and criminal sentencing recommendations (Tashea, 2017). In this paper we focus on the specific problem of gender bias, and analyse the extent to which it persists in modern TBTL models. We build upon Zhao et al. (2018a, 2019), in which quantification and mitigation of bias in ELMo was centre stage. In addressing this problem for more recent models, we aim to answer three main questions: i) How can we quantify bias in pre-trained language models? ii) How do different models compare in terms of bias? iii) How to mitigate bias in these models? We believe …

Content truncated to 500 words