Leveraging Pre-trained Language Models for Gender Debiasing

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 2188–2195 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 2188 Leveraging Pre-trained Language Models for Gender Debiasing Nishtha Jain1, Maja Popovic1, Declan Groves2, Lucia Specia3 1ADAPT Centre, 2Microsoft, 3Imperial College London 1Trinity College Dublin, Ireland, 1Dublin City University, Ireland, 2Dublin, Ireland, 3London, UK 1{firstname.lastname}@adaptcentre.ie, 2declan.groves@microsoft.com, 3l.specia@imperial.ac.uk Abstract Studying and mitigating gender and other biases in natural language have become important areas of research from both algorithmic and data perspectives. This paper explores the idea of reducing gender bias in a language generation context by generating gender variants of sentences. Previous work in this field has either been rule-based or required large amounts of gender balanced training data. These approaches are however not scalable across multiple languages, as creating data or rules for each language is costly and time-consuming. This work explores a light-weight method to generate gender variants for a given text using pre-trained language models as the resource, without any task-specific labelled data. The approach is designed to work on multiple languages with minimal changes in the form of heuristics. To showcase that, we have tested it on a high-resourced language, namely Spanish, and a low-resourced language from a different family, namely Serbian. The approach proved to work very well on Spanish, and while the results were less positive for Serbian, it showed potential even for languages where pre-trained models are less effective. Keywords: gender debiasing, language generation, pre-trained language models 1. Introduction Gender bias in language has increasingly become an important topic of research in natural language pro- cessing (NLP). Although NLP models are successful in modelling various applications, they propagate and may even amplify gender biases found in the training sets. While the study of bias in artificial intelligence is not new, techniques to mitigate gender bias in NLP are relatively nascent (Sun et al., 2019). This paper pro- poses a new method to reduce gender bias by enriching existing data with gender variants. These variants can be used either directly, for example, in the context of language generation applications to provide multiple gender-marked outputs, or to create gender-balanced corpora that can in turn be used as training data for NLP models. More specifically, the approach is devised to process either original or automatically generated sentences in gender-marked languages such as Spanish, where grammatical gender is expressed by morphology (see Section 3) to provide its gender variants. The method does not address only gender of persons but all in- stances of grammatical gender (persons, animals, ob- jects). For example, given the sentence in Spanish “¡Gracias, querida!” (“Thank you, darling!”), which uses the feminine noun “querida”, the goal is to gen- erate the masculine counterpart: “¡Gracias, querido!”. Conversely, given the sentence (also in Spanish) “En- viado.” (“Sent.”), which uses the masculine version of the past-participle verb “send”, the goal is to generate its feminine counterpart “Enviada.”. Our approach is inspired by work in the area of text in- filling (Zhu …

Content truncated to 500 words