Gender in Danger? Evaluating Speech Translation Technology on the MuST-SHE Corpus

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6923–6933 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 6923 Gender in Danger? Evaluating Speech Translation Technology on the MuST-SHE Corpus Luisa Bentivogli1∗, Beatrice Savoldi2∗, Matteo Negri1, Mattia Antonino Di Gangi1,2, Roldano Cattoni1, Marco Turchi1 1Fondazione Bruno Kessler 2University of Trento Trento, Italy {bentivo,negri,digangi,cattoni,turchi}@fbk.eu beatrice.savoldi@unitn.it Abstract Translating from languages without produc- tive grammatical gender like English into gender-marked languages is a well-known dif- ficulty for machines. This difficulty is also due to the fact that the training data on which mod- els are built typically reflect the asymmetries of natural languages, gender bias included. Ex- clusively fed with textual data, machine trans- lation is intrinsically constrained by the fact that the input sentence does not always con- tain clues about the gender identity of the re- ferred human entities. But what happens with speech translation, where the input is an au- dio signal? Can audio provide additional infor- mation to reduce gender bias? We present the first thorough investigation of gender bias in speech translation, contributing with: i) the re- lease of a benchmark useful for future studies, and ii) the comparison of different technolo- gies (cascade and end-to-end) on two language directions (English-Italian/French). 1 Introduction With the exponential popularity of deep learning ap- proaches for a great range of natural language pro- cessing (NLP) tasks being integrated in our daily life, the need to address the issues of gender fair- ness1 and gender bias has become a growing inter- disciplinary concern. Present-day studies on a vari- ety of NLP-related tasks, such as sentiment analysis (Kiritchenko and Mohammad, 2018) coreference resolution (Rudinger et al., 2018; Webster et al., 2018; Zhao et al., 2018), visual semantic-role label- ing (Zhao et al., 2017) or language modeling (Lu ∗∗These authors contributed equally. The work by Beatrice Savoldi was carried out during an internship at Fondazione Bruno Kessler. 1We acknowledge that gender is a multifaceted notion, not necessarily constrained within binary assumptions. However, since speech translation is hindered by the scarcity of available data, we rely on the female/male distinction of gender, as it is linguistically reflected in existing natural data. et al., 2019), attest the existence of a systemic bias that reproduces gender stereotypes discriminating women. In translation-related tasks, gender bias arises from the extent through which each language formally expresses the female or male gender of a referred human entity. Languages with a grammat- ical system of gender, such as Romance languages, rely on a copious set of morphological (inflection) and syntactic (gender agreement) devices apply- ing to numerous parts of speech (Hockett, 1958). Differently, English is a natural gender language that only reflects distinction of sex via pronouns, inherently gendered words (boy, girl) and excep- tionally with marked nouns (actor, actress). For all the other indistinct neutral words, the gender of the referred entity – if available – is inferred from contextual information present in the discourse, e.g. he/she is a friend. Nascent inquiries on machine …

Content truncated to 500 words