Under the Morphosyntactic Lens: A Multifaceted Evaluation of Gender Bias in Speech Translation

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 1807 - 1824 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Under the Morphosyntactic Lens: A Multifaceted Evaluation of Gender Bias in Speech Translation Beatrice Savoldi1,2, Marco Gaido1,2, Luisa Bentivogli2, Matteo Negri2, Marco Turchi2 1 University of Trento 2 Fondazione Bruno Kessler beatrice.savoldi@unitn.it {mgaido,bentivo,negri,turchi}@fbk.eu Abstract Gender bias is largely recognized as a prob- lematic phenomenon affecting language tech- nologies, with recent studies underscoring that it might surface differently across languages. However, most of current evaluation practices adopt a word-level focus on a narrow set of occupational nouns under synthetic condi- tions. Such protocols overlook key features of grammatical gender languages, which are characterized by morphosyntactic chains of gender agreement, marked on a variety of lexical items and parts-of-speech (POS). To overcome this limitation, we enrich the natu- ral, gender-sensitive MuST-SHE corpus (Ben- tivogli et al., 2020) with two new linguistic an- notation layers (POS and agreement chains), and explore to what extent different lexical categories and agreement phenomena are im- pacted by gender skews. Focusing on speech translation, we conduct a multifaceted evalu- ation on three language directions (English- French/Italian/Spanish), with models trained on varying amounts of data and different word segmentation techniques. By shedding light on model behaviours, gender bias, and its detec- tion at several levels of granularity, our findings emphasize the value of dedicated analyses be- yond aggregated overall results. 1 Introduction As Matasovi´c (2004) posits: “Gender is perhaps the only grammatical category that ever evoked passion – and not only among linguists.” That is because, in the case of human entities, masculine or feminine inflections are assigned semantically, i.e. in relation to the extra-linguistic reality of gender (Ackerman, 2019; Corbett, 1991, 2013). Thus, gen- dered features interact with the – sociocultural and political – perception and representation of individ- uals (Gygax et al., 2019), by prompting discussions on the appropriate recognition of gender groups and their linguistic visibility (Stahlberg et al., 2007; Hellinger and Motschenbacher, 2015; Hord, 2016). Such concerns also invested language technologies (Sun et al., 2019; Cao and Daumé III, 2020), where it has been shown that automatic translation sys- tems tend to over-represent masculine forms and amplify stereotypes when translating into grammat- ical gender languages (Savoldi et al., 2021). Current evaluation practices for assessing gen- der bias in both Machine (MT) and Speech Trans- lation (ST) commonly inspect such concerning behaviours by focusing only on a restricted set of occupational nouns (e.g. nurse, doctor), and on synthetic benchmarks (Stanovsky et al., 2019; Escudé Font and Costa-jussà, 2019; Renduchin- tala et al., 2021). Also, even when relying on lexically richer natural benchmarks, the designed metrics still work at the word level, treating all gender-marked words indiscriminately (Alhafni et al., 2020; Bentivogli et al., 2020). Accordingly, current test sets and protocols: i) do not allow us to inspect if and to what extent different word cat- egories participate in gender bias, ii) overlook the underlying morphosyntactic nature of …

Content truncated to 500 words