How Gender Debiasing Affects Internal Model Representations, and Why It Matters

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2602 - 2628 July 10-15, 2022 ©2022 Association for Computational Linguistics How Gender Debiasing Affects Internal Model Representations, and Why It Matters Hadas Orgad1 Seraphina Goldfarb-Tarrant2 Yonatan Belinkov1∗ 1Technion – Israel Institute of Technology 2University of Edinburgh orgad.hadas@cs.technion.ac.il s.tarrant@ed.ac.uk belinkov@technion.ac.il Abstract Common studies of gender bias in NLP focus either on extrinsic bias measured by model per- formance on a downstream task or on intrinsic bias found in models’ internal representations. However, the relationship between extrinsic and intrinsic bias is relatively unknown. In this work, we illuminate this relationship by measuring both quantities together: we debias a model during downstream fine-tuning, which reduces extrinsic bias, and measure the effect on intrinsic bias, which is operationalized as bias extractability with information-theoretic probing. Through experiments on two tasks and multiple bias metrics, we show that our intrinsic bias metric is a better indicator of de- biasing than (a contextual adaptation of) the standard WEAT metric, and can also expose cases of superficial debiasing. Our framework provides a comprehensive perspective on bias in NLP models, which can be applied to deploy NLP systems in a more informed manner. 1 1 Introduction Efforts to identify and mitigate gender bias in Nat- ural Language Processing (NLP) systems typically target one of two notions of bias. Extrinsic evalua- tion methods and debiasing techniques focus on the bias reflected in a downstream task (De-Arteaga et al., 2019; Zhao et al., 2018), while intrinsic methods focus on a model’s internal representa- tions, such as word or sentence embedding geom- etry (Caliskan et al., 2017; Bolukbasi et al., 2016; Guo and Caliskan, 2021). Despite an abundance of evidence pointing towards gender bias in pre- trained language models (LMs), the extent of harm caused by these biases is not clear when it is not reflected in a specific downstream task (Barocas ∗Supported by the Viterbi Fellowship in the Center for Computer Engineering at the Technion. 1Our code and model checkpoints are publicly avail- able at https://github.com/technion-cs-nlp/ gender_internal RoBERTa Dataset Classifier End-to-end finetuning RoBERTa Debiased Dataset Classifier End-to-end finetuning RoBERTa Probing Dataset Probe Classifier Probe training RoBERTa Probe Classifier (Debiased) Probe training Compression (intrinsic bias) Extrinsic bias Extrinsic bias (a) Training a model on the original task’s dataset (b) Training another model on a debiased dataset Compression (intrinsic bias) (c) Measuring our intrinsic metric on the debiased and original dataset using probing Figure 1: Our proposed framework. Black arrows mark forward passes, red arrows mark things we measure. We first (a) train a model on a downstream task, then (b) train another model on the same task using a debiased dataset, and finally (c) measure intrinsic bias in both models and compare. et al., 2017; Kate Crawford, 2017; Blodgett et al., 2020; Bommasani et al., 2021). For instance, while the word embedding proximity of “doctor” to “man” and “nurse” to “woman” is intuitively normatively wrong, it is not clear when such …

Content truncated to 500 words