Exploring Human Gender Stereotypes with Word Association Test

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 6133–6143, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 6133 Exploring Human Gender Stereotypes with Word Association Test Yupei Du , Yuanbin Wu , and Man Lan Department of Computer Science and Technology, East China Normal University {yupeidu.cs}@gmail.com {ybwu,mlan}@cs.ecnu.edu.cn Abstract Word embeddings have been widely used to study gender stereotypes in texts. One key problem regarding existing bias scores is to evaluate their validities: do they really re- flect true bias levels? For a small set of words (e.g. occupations), we can rely on hu- man annotations or external data. However, for most words, evaluating the correctness of them is still an open problem. In this work, we utilize word association test, which con- tains rich types of word connections anno- tated by human participants, to explore how gender stereotypes spread within our minds. Specifically, we use random walk on word association graph to derive bias scores for a large amount of words. Experiments show that these bias scores correlate well with bias in the real world. More importantly, compar- ing with word-embedding-based bias scores, it provides a different perspective on gender stereotypes in words.1 1 Introduction Consciously or not, gender stereotypes can be found in many places in our life. From propor- tions of female in certain occupations, to students’ choices on their majors, from the stories people been told in books and movies, to the decisions they make every day, researchers try to build dif- ferent stereotype models, to understand patterns behind various gender-related social and psycho- logical phenomena. Here, we investigate gender bias using the lens of language, especially, the words. Recent studies have shown that, by learning from large amounts of texts, low dimensional word embed- dings (Mikolov et al., 2013) can exhibit human gender bias: the vector of “engineer” is more close 1Our code is publicly available at https://github. com/Yupei-Du/bias-in-wat. to “he” than “she” (Bolukbasi et al., 2016). What’s more insightful is that the revealed bias correlates well with census data, and its trends on historical texts matches women’s movements in 1960-1970s (Garg et al., 2017). Thus word embeddings might be a reasonable tool in the study of gender stereo- types. In this work, we try to resolve two prob- lems regarding existing word-embedding-based bias scores. First, most embeddings are trained on word co-occurrence information in texts: “engi- neer” co-occurs with “he” more often than “she” in corpus. Since co-occurrence is only one of the various relations among words, it is natural to ask whether other types of relations can also re- flect gender bias. Second, the validity of word- embedding-based bias scores are often examined using a small set of occupation words (i.e., how they correlate with occupation statistics in the real world) or a small set of human annotated words. How to evaluate validities of bias scores on the remaining large amount of words is still an …

Content truncated to 500 words