Are Pretrained Multilingual Models Equally Fair across Languages?

Proceedings of the 29th International Conference on Computational Linguistics, pages 3597–3605 October 12–17, 2022. 3597 Are Pretrained Multilingual Models Equally Fair Across Languages? Laura Cabello Piqueras University of Copenhagen lcp@di.ku.dk Anders Søgaard University of Copenhagen soegaard@di.ku.dk Abstract Pretrained multilingual language models can help bridge the digital language divide, en- abling high-quality NLP models for lower- resourced languages. Studies of multilingual models have so far focused on performance, consistency, and cross-lingual generalisation. However, with their wide-spread application in the wild and downstream societal impact, it is important to put multilingual models un- der the same scrutiny as monolingual mod- els. This work investigates the group fairness of multilingual models, asking whether these models are equally fair across languages. To this end, we create a new four-way multilin- gual dataset of parallel cloze test examples (MozArt), equipped with demographic infor- mation (balanced with regard to gender and native tongue) about the test participants. We evaluate three multilingual models on MozArt – mBERT, XLM-R, and mT5 – and show that across the four target languages, the three mod- els exhibit different levels of group disparity, e.g., exhibiting near-equal risk for Spanish, but high levels of disparity for German. 1 Introduction Fill-in-the-gap cloze tests (Taylor, 1953) ask lan- guage learners to predict what words were removed from a text and it is a “procedure for measuring the effectiveness of communication”. Today, language models are trained to do the same (Devlin et al., 2019). This has the advantage that we can now use fill-in-the-gap cloze tests to directly compare the linguistic preferences of humans and language models, e.g., to investigate task-independent soci- olectal biases (group disparities) in language mod- els (Zhang et al., 2021). This paper presents a novel four-way parallel cloze dataset for English, French, German, and Spanish that enables apples-to-apples comparison across languages of group disparities in multilingual language models.1 1The language selection was given to us, because we rely on an existing word alignment dataset; see §2. EN ES DE FR WordPiece (avg. #tokens) 19.7 22.0 23.6 23.1 SentencePiece (avg. #tokens) 22.3 22.9 24.9 25.3 #Sentences 100 100 100 100 #Annotations 600 600 600 600 #Annotators 60 60 60 60 Demographics id_u, id_s, gender, age, nationality, first language, fluent languages, current country of residence, country of birth, time taken Table 1: MozArt details. The average number of to- kens per sentence is reported using WordPiece and Sen- tecePiece. The bottom row lists the demographic at- tributes shared; id_u refers to user id (anonymised) and id_s to sentence id. Language models induced from historical data are prone to implicit biases (Zhao et al., 2017; Chang et al., 2019; Mehrabi et al., 2021), e.g., as a result of the over-representation of male-dominated text sources such as Wikipedia and newswire (Hovy and Søgaard, 2015). This may lead to language models that are unfair to groups of users in the sense that they work better for some groups rather than others (Zhang et al., 2021). Multilingual lan- guage models can be unfair to their training lan- …

Content truncated to 500 words