On the Generalization of Projection-Based Gender Debiasing in Word Embedding

Proceedings of Recent Advances in Natural Language Processing, pages 336–343 Varna, Sep 4–6, 2023 https://doi.org/10.26615/978-954-452-092-2_038 336 On the Generalization of Projection-Based Gender Debiasing in Word Embedding Elisabetta Fersini, Antonio Candelieri, Lorenzo Pastore University of Milano-Bicocca Milan - Italy {elisabetta.fersini, antonio.candelieri}@unimib.it l.pastore6@campus.unimib.it Abstract Gender bias estimation and mitigation tech- niques in word embeddings lack an understand- ing of their generalization capabilities. In this work, we complement prior research by com- paring in a systematic way four gender bias met- rics (Word Embedding Association Test, Rela- tive Negative Sentiment Bias, Embedding Co- herence Test and Bias Analogy Test), two types of projection-based gender mitigation strategies (hard- and soft-debiasing) on three well-known word embedding representations (Word2Vec, FastText and Glove). The experiments have shown that the considered word embeddings are consistent between them but the debiasing techniques are inconsistent across the different metrics, also highlighting the potential risk of unintended bias after the mitigation strategies. 1 Introduction A recent body of work in Natural Language Pro- cessing (NLP) has focused attention on quantifying different types of bias through various approaches, spanning from psychological tests and performance differences for various tasks to the geometry of vec- tor spaces (Sun et al., 2019). Defining the type of bias is essential to estimate and mitigate it. Sev- eral forms of biases specific to NLP application have been introduced in the literature during the last 5 years (Nozza et al., 2019; Nissim et al., 2020; Goldfarb-Tarrant et al., 2021). In (Hitti et al., 2019) the authors defined gender bias in a text as the use of words or syntactic constructs that connote or imply an inclination or prejudice against one gender, highlighting that gender bias can evidence itself structurally, contextually, or in both forms. Structural bias occurs when the construction of sen- tences shows patterns closely tied to the presence of gender bias. On the other hand, contextual bias can happen in the tone, words, or context of a sen- tence. Unlike structural bias, this type of bias is not evident in grammatical structure but requires contextual background information and human per- ception. Therefore, gender bias can be discovered using both linguistic and extra-linguistic cues and can manifest itself in subtle or explicit ways, with differing degrees of intensity (Stanczak and Augen- stein, 2021; Caliskan et al., 2022; Sen et al., 2022). Furthermore, gender bias can easily propagate to models and downstream tasks, causing harm to the end-users (Bolukbasi et al., 2016). These forms of bias can emerge as representational harms and gender gaps. The current literature about gender bias estima- tion and mitigation related to word embeddings lacks an understanding of their generalization ca- pabilities. Therefore, this work complements prior research by providing the first systematic evidence on the generalization of estimating gender bias and debiasing techniques, including comprehen- sive quantitative and qualitative analyses. In partic- ular, we compared in a systematic way four gender bias metrics (Word Embedding Association Test (Caliskan et al., 2017), Relative Negative Sentiment Bias (Sweeney and Najafian, 2019), Embedding Coherence Test (Dev …

Content truncated to 500 words