Gender Representation in Open Source Speech Resources

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 6599–6605 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 6599 Gender Representation in Open Source Speech Resources Mahault Garnerin, Solange Rossato, Laurent Besacier LIG, Univ. Grenoble Alpes, CNRS, Grenoble INP, FR-38000 Grenoble, France firstname.lastname@univ-grenoble-alpes.fr Abstract With the rise of artificial intelligence (AI) and the growing use of deep-learning architectures, the question of ethics, transparency and fairness of AI systems has become a central concern within the research community. We address transparency and fairness in spoken language systems by proposing a study about gender representation in speech resources available through the Open Speech and Language Resource platform. We show that finding gender information in open source corpora is not straightforward and that gender balance depends on other corpus characteristics (elicited/non elicited speech, low/high resource language, speech task targeted). The paper ends with recommendations about metadata and gender information for researchers in order to assure better transparency of the speech systems built using such corpora. Keywords: speech resources, gender, metadata, open speech language resources (OpenSLR) 1. Introduction The ever growing use of machine learning has put data at the center of the industrial and research spheres. Indeed, for a system to learn how to associate an input X to an output Y, many paired examples are needed to learn this mapping process. This need for data coupled with the improvement in computing power and algorithm efficiency has led to the era of big data. But data is not only needed in mass, but also with a certain level of quality. In this paper we argue that one of the main quality of data is its transparency. In recent years, concerns have been raised about the bi- ases existing in the systems. A well-known case in Natural Language Processing (NLP) is the example of word em- beddings, with the studies of Bolukbasi et al. (2016) and Caliskan et al. (2017) which showed that data are socially constructed and hence encapsulate a handful of social rep- resentations and power structures, such as gender stereo- types. Gender-bias has also been found in machine trans- lation tasks (Vanmassenhove et al., 2018), as well as facial recognition (Buolamwini and Gebru, 2018) and is now at the center of research debates. In previous work, we inves- tigated the impact of gender imbalance in training data on the performance of an automatic speech recognition (ASR) system, showing that the under-representation of women led to a performance bias of the system for female speakers (Garnerin et al., 2019). In this paper, we survey the gender representation within an open platform gathering speech and language resources to develop speech processing tools. The aim of this survey is twofold: firstly, we investigate the gender balance within speech corpora in terms of speaker representation but also in terms of speech time available for each gender category. Secondly we propose a reflection about general practices when releasing resources, basing ourselves on some rec- ommendations from previous …

Content truncated to 500 words