Gender Bias in Multilingual Embeddings and Cross-Lingual Transfer

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2896–2907 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 2896 Gender Bias in Multilingual Embeddings and Cross-Lingual Transfer Jieyu Zhao§ ∗ Subhabrata Mukherjee‡ Saghar Hosseini‡ Kai-Wei Chang§ Ahmed Hassan Awadallah‡ §University of California, Los Angeles ‡Microsoft Research AI {jyzhao, kwchang}@cs.ucla.edu {Subhabrata.Mukherjee, Saghar.Hosseini, hassanam}@microsoft.com Abstract Multilingual representations embed words from many languages into a single semantic space such that words with similar meanings are close to each other regardless of the lan- guage. These embeddings have been widely used in various settings, such as cross-lingual transfer, where a natural language processing (NLP) model trained on one language is de- ployed to another language. While the cross- lingual transfer techniques are powerful, they carry gender bias from the source to target lan- guages. In this paper, we study gender bias in multilingual embeddings and how it affects transfer learning for NLP applications. We cre- ate a multilingual dataset for bias analysis and propose several ways for quantifying bias in multilingual representations from both the in- trinsic and extrinsic perspectives. Experimen- tal results show that the magnitude of bias in the multilingual representations changes differ- ently when we align the embeddings to dif- ferent target spaces and that the alignment di- rection can also have an influence on the bias in transfer learning. We further provide rec- ommendations for using the multilingual word representations for downstream tasks. 1 Introduction Natural Language Processing (NLP) plays a vital role in applications used in our daily lives. Despite the great performance inspired by the advanced machine learning techniques and large available datasets, there are potential societal biases em- bedded in these NLP tasks – where the systems learn inappropriate correlations between the final predictions and sensitive attributes such as gen- der and race. For example, Zhao et al. (2018a) and Rudinger et al. (2018) demonstrate that coref- erence resolution systems perform unequally on ∗Most of the work was done while the first author was an intern at Microsoft Research. different gender groups. Other studies show that such bias is exhibited in various components of the NLP systems, such as the training dataset (Zhao et al., 2018a; Rudinger et al., 2018), the embed- dings (Bolukbasi et al., 2016; Caliskan et al., 2017; Zhou et al., 2019; Manzini et al., 2019) as well as the pre-trained models (Zhao et al., 2019; Kurita et al., 2019). Recent advances in NLP require large amounts of training data. Such data may be available for resource-rich languages such as English, but they are typically absent for many other languages. Mul- tilingual word embeddings align the embeddings from various languages to the same shared em- bedding space which enables transfer learning by training the model in one language and adopting it for another one (Ammar et al., 2016; Ahmad et al., 2019b; Meng et al., 2019; Chen et al., 2019). Previous work has proposed different methods to create multilingual word embeddings. One com- mon way is to first …

Content truncated to 500 words