MABEL: Attenuating Gender Bias using Textual Entailment Data

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9681 - 9702 December 7-11, 2022 ©2022 Association for Computational Linguistics MABEL: Attenuating Gender Bias using Textual Entailment Data Jacqueline He∗ Mengzhou Xia Christiane Fellbaum Danqi Chen Department of Computer Science, Princeton University jacquelinehe00@gmail.com {mengzhou, fellbaum, danqic}@cs.princeton.edu Abstract Pre-trained language models encode undesir- able social biases, which are further exacer- bated in downstream use. To this end, we pro- pose MABEL (a Method for Attenuating Gen- der Bias using Entailment Labels), an interme- diate pre-training approach for mitigating gen- der bias in contextualized representations. Key to our approach is the use of a contrastive learn- ing objective on counterfactually augmented, gender-balanced entailment pairs from natural language inference (NLI) datasets. We also introduce an alignment regularizer that pulls identical entailment pairs along opposite gen- der directions closer. We extensively evaluate our approach on intrinsic and extrinsic metrics, and show that MABEL outperforms previous task-agnostic debiasing approaches in terms of fairness. It also preserves task performance af- ter fine-tuning on downstream tasks. Together, these findings demonstrate the suitability of NLI data as an effective means of bias mitiga- tion, as opposed to only using unlabeled sen- tences in the literature. Finally, we identify that existing approaches often use evaluation set- tings that are insufficient or inconsistent. We make an effort to reproduce and compare previ- ous methods, and call for unifying the evalua- tion settings across gender debiasing methods for better future comparison.1 1 Introduction Pre-trained language models have reshaped the landscape of modern natural language process- ing (Peters et al., 2018; Devlin et al., 2019; Liu et al., 2019). As these powerful networks are opti- mized to learn statistical properties from large train- ing corpora imbued with significant social biases (e.g., gender, racial), they produce encoded repre- sentations that inherit undesirable associations as a ∗This work was done before JH graduated from Princeton University. 1Our code is publicly available at https://github.com/ princeton-nlp/MABEL. byproduct (Zhao et al., 2019; Webster et al., 2020; Nadeem et al., 2021). More concerningly, models trained on these representations can not only prop- agate but also amplify discriminatory judgments in downstream applications (Kurita et al., 2019). A multitude of recent efforts have focused on alleviating biases in language models. These can be classed into two categories (Table 1): 1) task- specific approaches perform bias mitigation during downstream fine-tuning, and require data to be an- notated for sensitive attributes; 2) task-agnostic approaches directly improve pre-trained representa- tions, most commonly either by removing discrim- inative biases through projection (Dev et al., 2020; Liang et al., 2020; Kaneko and Bollegala, 2021), or by performing intermediate pre-training on gender- balanced data (Webster et al., 2020; Cheng et al., 2021; Lauscher et al., 2021; Guo et al., 2022), re- sulting in a new encoder that transfers fairness ef- fects downstream via standard fine-tuning. In this work, we present MABEL, a novel and lightweight method for attenuating gender bias. MABEL is task-agnostic and can be framed as an intermediate pre-training approach …

Content truncated to 500 words