Neutralizing Gender Bias in Word Embeddings with Latent Disentanglement and Counterfactual Generation

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3126–3140 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 3126 Neutralizing Gender Bias in Word Embeddings with Latent Disentanglement and Counterfactual Generation Seungjae Shin, Kyungwoo Song, JoonHo Jang, Hyemi Kim, Weonyoung Joo, Il-Chul Moon Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Korea {tmdwo0910,gtshs2,adkto8093,khm0308,es345,icmoon}@kaist.ac.kr Abstract Recent research demonstrates that word em- beddings, trained on the human-generated cor- pus, have strong gender biases in embedding spaces, and these biases can result in the discriminative results from the various down- stream tasks. Whereas the previous methods project word embeddings into a linear sub- space for debiasing, we introduce a Latent Disentanglement method with a siamese auto- encoder structure with an adapted gradient re- versal layer. Our structure enables the sepa- ration of the semantic latent information and gender latent information of given word into the disjoint latent dimensions. Afterwards, we introduce a Counterfactual Generation to con- vert the gender information of words, so the original and the modified embeddings can pro- duce a gender-neutralized word embedding af- ter geometric alignment regularization, with- out loss of semantic information. From the various quantitative and qualitative debiasing experiments, our method shows to be better than existing debiasing methods in debiasing word embeddings. In addition, Our method shows the ability to preserve semantic infor- mation during debiasing by minimizing the se- mantic information losses for extrinsic NLP downstream tasks. 1 Introduction Recent researches have disclosed that word embed- dings contain unexpected bias in their geometry on the embedding space (Bolukbasi et al., 2016; Zhao et al., 2019). The bias reflects unwanted stereo- types such as the correlation between gender1 and occupation words. Bolukbasi et al. (2016) enumer- ated that the automatically generated analogies of (she, he) in the Word2Vec (Mikolov et al., 2013b) show the gender biases in significant level. An 1While we acknowledge a potential and expanded defini- tion on gender as stated in Larson (2017), we only cover the gender bias between the male and female in this paper. (𝑎𝑎) (𝑏𝑏) (𝑐𝑐) (𝑑𝑑) Geometric Alignment : Feminine (green) word embedding and Masculine (orange) word embedding with gender-pair relationship : Gender biased (purple) word embedding, gender-counterfactual (red) word embedding, and Neutralized (gray) word embedding Figure 1: The process view of our method. We can improve the embedding space from (a) to (b) with a better-aligned structure between gender word pairs by the proposed latent disentanglement. Afterwards, (c) we generate the gender-counterfactual embedding of the gender-biased word while keeping a geometrically aligned relationship with the gender word pairs to guar- antee that the pair of word embeddings only differs from gender information, not hurting semantic infor- mation. (d) We obtain the gender-neutralized word em- bedding by interpolating the embedding from the pair of original-counterfactual word embeddings. example of the analogies is the relatively closer dis- tance of she to nurse; and he to doctor. Garg et al. (2018) demonstrated that the embeddings, from Word2Vec (Mikolov et al., 2013a) to Glove (Pen- nington et …

Content truncated to 500 words