Learning Bias-reduced Word Embeddings Using Dictionary Definitions

Findings of the Association for Computational Linguistics: ACL 2022, pages 1139 - 1152 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Learning Bias-reduced Word Embeddings Using Dictionary Definitions Haozhe An∗ University of Maryland, College Park haozhe@umd.edu Xiaojiang Liu and Jian Zhang Apple {xiaojiang_liu, donald_zhang} @apple.com Abstract Pre-trained word embeddings, such as GloVe, have shown undesirable gender, racial, and re- ligious biases. To address this problem, we propose DD-GloVe, a train-time debiasing algo- rithm to learn word embeddings by leveraging dictionary definitions. We introduce dictionary- guided loss functions that encourage word em- beddings to be similar to their relatively neu- tral dictionary definition representations. Exist- ing debiasing algorithms typically need a pre- compiled list of seed words to represent the bias direction, along which biased information gets removed. Producing this list involves subjec- tive decisions and it might be difficult to obtain for some types of biases. We automate the pro- cess of finding seed words: our algorithm starts from a single pair of initial seed words and au- tomatically finds more words whose definitions display similar attributes traits. We demon- strate the effectiveness of our approach with benchmark evaluations and empirical analyses. Our code is available at https://github. com/haozhe-an/DD-GloVe. 1 Introduction Word embeddings can meaningfully capture seman- tic and syntactic similarities between words. Pop- ular embeddings are Word2Vec (Mikolov et al., 2013b), GloVe (Pennington et al., 2014), and Fast- Text (Bojanowski et al., 2017). Although contex- tual word embeddings, like BERT embeddings (De- vlin et al., 2019) and ELMo (Peters et al., 2018), gain increasing popularity, some recent research keeps using static word embeddings as input to their state-of-the-art algorithms in downstream nat- ural language processing and computer vision ap- plications (Guan et al., 2021; Gao et al., 2021). Despite the effectiveness of word embeddings, biases in them reflect undesirable association be- tween some concepts. Bolukbasi et al. (2016) first identify that the distance between −−→ man and ∗ Work done during an internship at Apple. Figure 1: Definitions of example gender-specific and gender-biased words. Gender-specific words typically contain gendered words in their definitions, whereas gender-biased words tend to have neutral definitions. −−−−→ woman is close to that between −−−−−−−−→ programmer and −−−−−−−→ homemaker. Similar phenomena in word embed- dings lead to biased interpretations in the word analogy task, associating certain words with gen- der, racial, and religious stereotypes (Manzini et al., 2019). Deploying such biased word embeddings in downstream tasks would cause allocational and representational harms (Blodgett et al., 2020). It is important to learn bias-reduced word embeddings. Dictionary definitions, however, are a neutral source for mitigating biases in word embeddings. The objective, impartial, and concise definitions of words in a dictionary could be unbiased refer- ence points. We propose to encourage word em- beddings to be similar to their relatively neutral representations in a dictionary for bias reduction. We simultaneously train and debias the word em- beddings from a new initialization point, so as to learn distributional representations and mitigate bi- ases using dictionary definitions concurrently. In addition, …

Content truncated to 500 words