Upstream Mitigation Is Not All You Need: Testing the Bias Transfer Hypothesis in Pre-Trained Language Models

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 3524 - 3542 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Upstream Mitigation Is Not All You Need: Testing the Bias Transfer Hypothesis in Pre-Trained Language Models Ryan Steed Carnegie Mellon University ryansteed@cmu.edu Swetasudha Panda, Ari Kobren, Michael Wick Oracle Labs {swetasudha.panda,ari.kobren,michael.wick}@oracle.com A few large, homogenous, pre-trained models undergird many machine learning systems — and often, these models contain harmful stereotypes learned from the internet. We investigate the bias transfer hypothesis: the theory that social biases (such as stereotypes) internalized by large language models during pre-training transfer into harmful task-specific behavior after fine-tuning. For two classification tasks, we find that reducing intrin- sic bias with controlled interventions before fine- tuning does little to mitigate the classifier’s dis- criminatory behavior after fine-tuning. Regression analysis suggests that downstream disparities are better explained by biases in the fine-tuning dataset. Still, pre-training plays a role: simple alterations to co-occurrence rates in the fine-tuning dataset are ineffective when the model has been pre-trained. Our results encourage practitioners to focus more on dataset quality and context-specific harms. 1 Introduction Large language models (LLMs) and other mas- sively pre-trained “foundation” models are power- ful tools for task-specific machine learning (Bom- masani et al., 2021). Models pre-trained by well- resourced organizations can easily adapt to a wide variety of downstream tasks in a process called fine- tuning. But massive pre-training datasets and in- creasingly homogeneous model design come with well-known, immediate social risks beyond the fi- nancial and environmental costs (Strubell et al., 2019; Bender et al., 2021). Transformer-based LLMs like BERT and GPT- 3 contain quantifiable intrinsic social biases en- coded in their embedding spaces (Goldfarb-Tarrant et al., 2021). These intrinsic biases are typically associated with representational harms, including stereotyping and denigration (Barocas et al., 2017; Blodgett et al., 2020; Bender et al., 2021). Sepa- rately, many studies document the extrinsic harms of the downstream (fine-tuned & task-specific) ap- ���������� ��������������� ����������������� ���������������� ������������ ����������� ������������ ������� ����������������� ������������� �������� ������������ ����������� ������������ ���������� ���������� ���������������������� ����������������������� ���������������������� ��������������� ��������� ���������� ��������� �������� ���������� ������������� Figure 1: Full pre-training to fine-tuning pipeline, with experimental interventions (green hexagons). plications of fine-tuned LLMs, including discrimi- natory medical diagnoses (Zhang et al., 2020), over- reliance on binary gender for coreference resolu- tion (Cao and Daumé, 2021), the re-inforcement of traditional gender roles in part-of-speech tag- ging (Garimella et al., 2019), toxic text generation (Gehman et al., 2020), and censorship of inclu- sive language and AAVE (Blodgett and O’Connor, 2017; Blodgett et al., 2018; Park et al., 2018; Sap et al., 2019). Despite these risks, no research has investigated the extent to which downstream systems inherit social biases from pre-trained models.1 Many stud- 1We use the term “bias” to refer to statistical associations 3524 ies warn that increasing intrinsic bias upstream may lead to an increased risk of downstream harms (Bolukbasi et al., 2016; Caliskan et al., 2017). This hypothesis, which we call the …

Content truncated to 500 words