Evaluating Gender Bias in Machine Translation

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1679–1684 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 1679 Evaluating Gender Bias in Machine Translation Gabriel Stanovsky1,2, Noah A. Smith1,2, and Luke Zettlemoyer1 1Paul G. Allen School of Computer Science & Engineering, University of Washington, Seattle, USA 2Allen Institute for Artificial Intelligence, Seattle, USA {gabis,nasmith,lsz}@cs.washington.edu Abstract We present the first challenge set and eval- uation protocol for the analysis of gender bias in machine translation (MT). Our ap- proach uses two recent coreference resolution datasets composed of English sentences which cast participants into non-stereotypical gender roles (e.g., “The doctor asked the nurse to help her in the operation”). We devise an automatic gender bias evaluation method for eight tar- get languages with grammatical gender, based on morphological analysis (e.g., the use of fe- male inflection for the word “doctor”). Our analyses show that four popular industrial MT systems and two recent state-of-the-art aca- demic MT models are significantly prone to gender-biased translation errors for all tested target languages. Our data and code are pub- licly available at https://github.com/ gabrielStanovsky/mt_gender. 1 Introduction Learned models exhibit social bias when their training data encode stereotypes not relevant for the task, but the correlations are picked up any- way. Notable examples include gender biases in visual SRL (cooking is stereotypically done by women, construction workers are stereotypically men; Zhao et al., 2017), lexical semantics (“man is to computer programmer as woman is to home- maker”; Bolukbasi et al., 2016), and natural lan- guage inference (associating women with gossip- ing and men with guitars; Rudinger et al., 2017). In this work, we conduct the first large-scale multilingual evaluation of gender-bias in machine translation (MT), following recent small-scale qualitative studies which observed that online MT services, such as Google Translate or Microsoft Translator, also exhibit biases, e.g., translating nurses as females and programmers as males, re- gardless of context (Alvarez-Melis and Jaakkola, The doctor asked the nurse to help her in the procedure El doctor le pidio a la enfermera que le ayudara con el procedimiento Figure 1: An example of gender bias in machine trans- lation from English (top) to Spanish (bottom). In the English source sentence, the nurse’s gender is un- known, while the coreference link with “her” identi- fies the “doctor” as a female. On the other hand, the Spanish target sentence uses morphological features for gender: “el doctor” (male), versus “la enfermer- a” (female). Aligning between source and target sen- tences reveals that a stereotypical assignment of gender roles changed the meaning of the translated sentence by changing the doctor’s gender. 2017; Font and Costa-Juss`a, 2019). Google Trans- late recently tried to mitigate these biases by al- lowing users to sometimes choose between gen- dered translations (Kuczmarski, 2018). As shown in Figure 1, we use data introduced by two recent coreference gender-bias studies: the Winogender (Rudinger et al., 2018), and the Wino- Bias (Zhao et al., 2018) datasets. Following the Winograd schema (Levesque, …

Content truncated to 500 words