Assessing Multilingual Fairness in Pre-trained Multimodal Representations

Findings of the Association for Computational Linguistics: ACL 2022, pages 2681 - 2695 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Assessing Multilingual Fairness in Pre-trained Multimodal Representations Jialu Wang and Yang Liu and Xin Eric Wang Department of Computer Science and Engineering University of California, Santa Cruz {faldict,yangliu,xwang366}@ucsc.edu Abstract Recently pre-trained multimodal models, such as CLIP (Radford et al., 2021), have shown ex- ceptional capabilities towards connecting im- ages and natural language. The textual repre- sentations in English can be desirably trans- ferred to multilingualism and support down- stream multimodal tasks for different lan- guages. Nevertheless, the principle of multi- lingual fairness is rarely scrutinized: do mul- tilingual multimodal models treat languages equally? Are their performances biased to- wards particular languages? To answer these questions, we view language as the fairness recipient and introduce two new fairness no- tions, multilingual individual fairness and mul- tilingual group fairness, for pre-trained mul- timodal models. Multilingual individual fair- ness requires that text snippets expressing sim- ilar semantics in different languages connect similarly to images, while multilingual group fairness requires equalized predictive perfor- mance across languages. We characterize the extent to which pre-trained multilingual vision- and-language representations are individually fair across languages. However, extensive ex- periments demonstrate that multilingual repre- sentations do not satisfy group fairness: (1) there is a severe multilingual accuracy dispar- ity issue; (2) the errors exhibit biases across languages conditioning the group of people in the images, including race, gender and age. 1 Introduction Recently pre-trained vision-and-language represen- tations (Lu et al., 2019; Tan and Bansal, 2019; Su et al., 2020; Li et al., 2020a; Chen et al., 2020; Li et al., 2020b; Gan et al., 2020; Yu et al., 2021; Desai and Johnson, 2021; Radford et al., 2021; Cho et al., 2021) have received a surge of atten- tion. Such pre-trained multimodal representations have shown great capabilities of bridging images and natural language on the downstream tasks, in- cluding image captioning (Laina et al., 2019), im- age retrieval (Vo et al., 2019), visual QA (Zhou et al., 2020), text-to-image generation (Ramesh et al., 2021), etc. While it is commonly recognized that the multimodal representations trained on En- glish corpora can be generalized to multilingualism by cross-lingual alignment (Lample and Conneau, 2019; Conneau et al., 2020), recent studies criti- cize that the multilingual textual representations do not learn equally high-quality representations for all the languages (Wu and Dredze, 2020), espe- cially for low-resource languages. Hu et al. (2020) emphasize the need for general-purpose represen- tations to seek equal performance across all lan- guages. However, there is still a lack of a nuanced understanding of how multilingual representations fare on vision-and-language benchmarks. This paper provides a novel perspective for an- alyzing the principles of multilingual fairness in multimodal representations from two aspects. First, existing frameworks for measuring multilingual biases usually emulate text sources in different lan- guages, which may have ambiguous meanings in varied contexts (González et al., 2020). In con- trast, we leverage visual grounding as the anchor to …

Content truncated to 500 words