DeBiasByUs: Raising Awareness and Creating a Database of MT Bias

DeBiasByUs: Raising Awareness and Creating a Database of MT Bias Joke Daems Postdoctoral Researcher @ LT3 UGent joke.daems@ugent.be Janiça Hackenbuchner Research Associate @ TH Köln janica.hackenbuchner@th-koeln.de Abstract This paper presents the project proposed by the DeBiasByUs1 team resulting from the Artificially Correct Hackathon. We briefly explain the hackathon challenge on 'Database and detection of gender bias in A.I. translations', highlight the importance of gender bias in Machine Translation (MT), describe our solution, the current status of the project, and our future plans. 1 Introduction to DeBiasByUs The DeBiasByUs project was a winning solution to a challenge on 'Creating Datasets and Resources against Societal Biases in AI2' at the Artificially Correct Hackathon organised by the Goethe-Institut in October 2021. The initial Hackathon team consisted of five participants3, of which the authors of the present paper will continue to develop the project. The goal of the challenge was to define and analyse gender bias from MT systems and either create a dataset or a platform for users to gather, describe, and discuss cases of bias. 2 The Problem of Gender Bias in MT MT systems are trained with data that contain biases present in our society and in our language. As such, these systems will reproduce or even heighten these biases, potentially leading to discrimination and harm. For example, translation datasets have a dominance of white male representation (Saunders and Byrne, 2020), and word embeddings (used to train MT systems) have been shown to reinforce gender stereotypes (Bolukbasi et al., 2016). © 2022 The authors. This article is licensed under a Crea- tive Commons 3.0 licence, no derivative works, attribu- tion,CC-BY-ND. 1 The original Hackathon project was “BiasByUs” but has now been changed to “DeBiasByUs” Different factors can contribute to bias in MT. There are linguistic factors, socio-cultural factors, reinforcement of historical gender stereotypes, especially in professions, and a lack of an explicit linguistic representation of nonbinary gender. While less apparent for genderless languages (e.g. Finnish) and notional gender languages (e.g. Danish), MT most often exhibits gender bias or opts for the generic masculine for grammatical gender languages (e.g. Spanish), where nouns, verbs, adjectives etc. carry gender inflections (Savoldi et al., 2021). Technical factors include MT sampling methods favoring masculine forms due to asymmetrical gender distributions in the training datasets, leading to reinforcement of gender stereotypes as the most common form is subsequently being chosen as a most-likely translation by the MT system (Shah et al., 2020). 3 Solution: Raising Awareness and Da- tabase Creation As a solution to the hackathon challenge, we created a website4 that serves a dual purpose: 1) raise public awareness about the issue of gender bias in MT by providing information and research findings, and 2) create a community-driven database of occurrences of gender bias in MT. The collected inputs can be moderated and reviewed by experts. Through such collaborative and community-driven action, we aim to create a database representing different language combinations that can then be used as biased test datasets for further research. The moderated …

Content truncated to 500 words