Gender bias Evaluation in Luganda-English Machine Translation

Gender Bias Evaluation in Luganda-English Machine Translation Eric Peter Wairagala kigaye.ericpeter@gmail.com Jonathan Mukiibi jonmuk7@gmail.com Jeremy Francis Tusubira tusubirafrancisjeremy@gmail.com Claire Babirye clarybits68@gmail.com Joyce Nakatumba-Nabende joyce.nabende@mak.ac.ug Department of Computer Science, Makerere University, Uganda, Kampala Andrew Katumba andrew.katumba@mak.ac.ug Department of Electrical and Computer Engineering, Makerere University, Uganda Kampala Ivan Ssenkungu ssenkungu.ivanandrew@gmail.com Department of Languages, Literature and Communication, Makerere University, Uganda Kam- pala Abstract We have seen significant growth in the area of building Natural Language Processing (NLP) tools for African languages. However, the evaluation of gender bias in the machine translation systems for African languages is not yet thoroughly investigated. This is due to the unavailabil- ity of explicit text data available for addressing the issue of gender bias in machine translation. In this paper, we use transfer learning techniques based on a pre-trained Marian MT model for building machine translation models for English-Luganda and Luganda-English. Our work at- tempts to evaluate and quantify the gender bias within a Luganda-English machine translation system using Word Embeddings Fairness Evaluation Framework (WEFE). Luganda is one of the languages with gender-neutral pronouns in the world, therefore we use a small set of trusted gendered examples as the test set to evaluate gender bias by biasing word embeddings. This approach allows us to focus on Luganda-English translations with gender-specific pronouns, and the results of the gender bias evaluation are confirmed by human evaluation. To compare and contrast the results of the word embeddings evaluation metric, we used a modified version of the existing Translation Gender Bias Index (TGBI) based on the grammatical consideration for Luganda. 1 Introduction Uganda is a highly multilingual country with over 43 known indigenous languages and dialects (Eberhard et al., 2020). However, many languages have no existing resources for building Nat- ural Language Processing (NLP) datasets and tools. One of the major languages spoken in Uganda is Luganda, primarily spoken in the South Eastern Buganda region, mainly along the shores of Lake Victoria and up north towards the Lake Kyoga shores (Nakayiza, 2013; Olaide and Azizi, 2019). Luganda is spoken by more than six million people, principally in central Uganda, including Kampala, the capital of Uganda. Topologically, it is a highly agglutinating, tonal language with subject-verb-object, word order, and nominative-accusative morphosyntac- tic alignment (Olaide and Azizi, 2019). Like many languages in sub-Saharan Africa, Luganda has limited text and speech data resources, making it a low-resourced language. However, due to the increase in the availability of computational resources and datasets, there has been high advancement in NLP research. Natural language processing applications approach tasks ranging from low-level processing, such as assigning parts of speech to words, to high-level tasks, such as question answering, and translating speech and text from one language to another. In this paper, we focus on building and evaluating Machine Translation (MT) models for Luganda as an application of natural language processing. Classically, rule-based systems were used for MT, these were replaced with statistical methods (Macketanz et al., 2017) in the 1990s. More recently, deep neural network models have achieved state-of-art …

Content truncated to 500 words