Evaluating Gender Bias in Speech Translation

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 2141–2147 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 2141 Evaluating Gender Bias in Speech Translation Marta R. Costa-juss`a1, Christine Basta1,2, Gerard I. G´allego1 1 TALP Research Center, Universitat Polit`ecnica de Catalunya, Barcelona, Spain 2 Institute of Graduate Studies and Research, Alexandria University, Egypt {marta.ruiz, christine.raouf.saad.basta, gerard.ion.gallego}@upc.edu Abstract The scientific community is increasingly aware of the necessity to embrace pluralism and consistently represent major and minor social groups. Currently, there are no standard evaluation techniques for different types of biases. Accordingly, there is an urgent need to provide evaluation sets and protocols to measure existing biases in our automatic systems. Evaluating the biases should be an essential step towards mitigating them in the systems. This paper introduces WinoST, a new freely available challenge set for evaluating gender bias in speech translation. WinoST is the speech version of WinoMT, an MT challenge set, and both follow an evaluation protocol to measure gender accuracy. Using an S-Transformer end-to-end speech translation system, we report the gender bias evaluation on four language pairs, and we reveal the inaccuracies in translations generating gender-stereotyped translations. Keywords: challenge set, multilinguality, direct speech-to-text translation 1. Introduction There is a massive lack of representation of diverse gender, race, cultural groups in the power structures. These problems permeate data science causing unbal- anced progress in this area, which misrepresents social groups by gender, race, and nationality. Biases have been shown in Machine Translation (MT) systems when translating from high-resource languages to low-resourced ones (Aharoni et al., 2019) and from less-morphological languages to high- morphological ones producing gender-stereotyped forms in inaccurate translations (Stanovsky et al., 2019; Prates et al., 2020; Saunders and Byrne, 2020a). Ad- ditionally, Automatic Speech Recognition (ASR) has demonstrated biases having a higher error rate for fe- male voices than for males (Tatman, 2017). Speech Translation (ST) is at the intersection of ASR and MT tasks, perpetuating biases from both tasks. While this is still an uncovered challenge in most ar- tificial intelligence tasks, the scientific community is making huge efforts towards bringing to light this chal- lenge. In the short term, researchers are devoting ef- forts towards bias evaluation, detection, and mitigation methods. On the other hand, there is multidisciplinary work in education, politics, and communications in the long-term perspective. For addressing this problem in MT, some papers pro- pose evaluating gender bias in MT (Stanovsky et al., 2019; Levy et al., 2021), creating multilingual balanced data to train fairer systems (Costa-juss`a et al., 2020), or using equalizing techniques to mitigate the effect of unbalanced training data (Font and Costa-juss`a, 2019; Saunders and Byrne, 2020a). Finally, in ST, (Ben- tivogli et al., 2020) have proposed the MuST-SHE cor- pus, a natural benchmark for automatically evaluating gender bias. The main contribution of our work is providing a large- scale multilingual ST challenge set,1 which follows an evaluation protocol for the analysis of gender bias, pre- viously proposed for MT (Stanovsky …

Content truncated to 500 words