Gender Bias in Masked Language Models for Multiple Languages

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2740 - 2750 July 10-15, 2022 ©2022 Association for Computational Linguistics Gender Bias in Masked Language Models for Multiple Languages Masahiro Kaneko1 Aizhan Imankulova2 Danushka Bollegala3,4∗ Naoaki Okazaki1 1Tokyo Institute of Technology 3University of Liverpool2CogSmart Co., Ltd. 4Amazon masahiro.kaneko@nlp.c.titech.ac.jp aizhan.imankulova@cogsmart-global.com danushka@liverpool.ac.uk okazaki@c.titech.ac.jp Abstract Masked Language Models (MLMs) pre-trained by predicting masked tokens on large corpora have been used successfully in natural language processing tasks for a variety of languages. Un- fortunately, it was reported that MLMs also learn discriminative biases regarding attributes such as gender and race. Because most stud- ies have focused on MLMs in English, the bias of MLMs in other languages has rarely been investigated. Manual annotation of eval- uation data for languages other than English has been challenging due to the cost and diffi- culty in recruiting annotators. Moreover, the existing bias evaluation methods require the stereotypical sentence pairs consisting of the same context with attribute words (e.g. He/She is a nurse). We propose Multilingual Bias Eval- uation (MBE) score, to evaluate bias in various languages using only English attribute word lists and parallel corpora between the target lan- guage and English without requiring manually annotated data. We evaluated MLMs in eight languages using the MBE and confirmed that gender-related biases are encoded in MLMs for all those languages. We manually created datasets for gender bias in Japanese and Rus- sian to evaluate the validity of the MBE. The results show that the bias scores reported by the MBE significantly correlates with that com- puted from the above manually created datasets and the existing English datasets for gender bias. 1 Introduction Masked Language Models (MLMs) (Devlin et al., 2019), which are pre-trained on large corpora, have been used successfully in natural language process- ing tasks for various languages (Conneau and Lam- ple, 2019; Martin et al., 2020; Conneau et al., 2020). Unfortunately, it has been reported that MLMs also ∗Danushka Bollegala holds concurrent appointments as a Professor at University of Liverpool and as an Amazon Scholar. This paper describes work performed at the Univer- sity of Liverpool and is not associated with Amazon. learn social biases regarding attributes such as gen- der, religion, and race (Kurita et al., 2019; Dev et al., 2020; Kaneko and Bollegala, 2021a; Bender et al., 2021). The bias in MLMs is evaluated by the imbalance of the likelihood between pairs of sentences associated with an attribute that has a common context (e.g. He/She is a nurse). Nadeem et al. (2021) masked the modified tokens (e.g. He, She), and Nangia et al. (2020) masked the unmod- ified tokens (e.g. is, a, nurse) one word at a time and calculated the likelihood from their predictions to evaluate the bias. Kaneko and Bollegala (2021c) evaluated the bias using the average of the likeli- hoods of all tokens without masking the MLM. Despite the numerous studies of social bias in MLMs covering English, social biases …

Content truncated to 500 words