Fairness in Language Models Beyond English: Gaps and Challenges

Findings of the Association for Computational Linguistics: EACL 2023, pages 2106–2119 May 2-6, 2023 ©2023 Association for Computational Linguistics Fairness in Language Models Beyond English: Gaps and Challenges Krithika Ramesh, Sunayana Sitaram, Monojit Choudhury Microsoft Corporation {t-kriramesh, sunayana.sitaram, monojitc}@microsoft.com Abstract With language models becoming increasingly ubiquitous, it has become essential to address their inequitable treatment of diverse demo- graphic groups and factors. Most research on evaluating and mitigating fairness harms has been concentrated on English, while multilin- gual models and non-English languages have received comparatively little attention. In this paper, we survey different aspects of fairness in languages beyond English and multilingual contexts. This paper presents a survey of fair- ness in multilingual and non-English contexts, highlighting the shortcomings of current re- search and the difficulties faced by methods designed for English. We contend that the mul- titude of diverse cultures and languages across the world makes it infeasible to achieve com- prehensive coverage in terms of constructing fairness datasets. Thus, the measurement and mitigation of biases must evolve beyond the cur- rent dataset-driven practices that are narrowly focused on specific dimensions and types of bi- ases and, therefore, impossible to scale across languages and cultures. 1 Introduction Language models are known to be susceptible to developing spurious correlations and encoding bi- ases that have potentially harmful consequences in downstream tasks. Whilst prior work has doc- umented these harms (Dev et al., 2021) (Bender et al., 2021) (Kumar et al.), there remains much to be studied and criticism for the existing research (or lack thereof) that remains to be addressed. In the context of language models, fairness can manifest in two forms; representational and allo- cational harms. Representational harms generally refer to cases where demographic groups end up be- ing misrepresented. This includes stereotypes and negative associations with these groups and even a lack of acknowledgment of certain groups that are underrepresented in the data. Allocational harms, on the other hand, refer to the inequitable distri- bution of resources and opportunities to groups with different demographic attributes associated with them. The nature of allocational harms can vary based on the sociocultural, economic, and legal settings where the system has been deployed. However, it can also take shape in terms of the model’s functionality across languages with fewer resources (Choudhury and Deshpande, 2021; Liu et al., 2021). While current literature adopts a Euro- American-centric view of fairness, work such as Sambasivan et al. (2021) pushes to recognize algo- rithmic fairness from a more inclusive lens. Bias crops up in multiple steps of the pipeline (Hovy and Prabhumoye, 2021) (Sap et al., 2022), including the annotation process, the training data, the input representations, model architecture, and the structure of the research design. Thus, mea- sures to mitigate bias in one of these components alone will likely not suffice as a corrective measure, necessitating human intervention at different stages of the pipeline. Most work that addresses fairness in NLP ad- dresses it from an Anglo-centric context, with comparatively significantly less work done in …

Content truncated to 500 words